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ARMY  MAINTENANCE  TRAINING  AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS)  DEVICE 
EVALUATION;  VOLUME  III,  QUALITATIVE  ASSESSMENT  OF  TWO  PROTOTYPE  DEVICES 


EXECUTIVE  SUMMARY 


Requirement: 

To  assess  and  document  user  opinions  about  the  usefulness  and  effective¬ 
ness  of  two  breadboard  AMTESS  devices. 


Procedure: 

Students,  instructors,  and  course  developers  from  several  maintenance 
MOS  and  locations  provided  opinion-based  data  concerning  the  effectiveness 
of  two  prototype  AMTESS  devices.  The  following  instruments  were  used  to 
collect  data:  initial  instructor  questionnaire  (completed  by  instructors 
prior  to  extensive  interaction  with  the  devices),  instructor  questionnaire 
(completed  by  instructors  after  they  had  used  the  devices  to  train  students), 
course  developer  questionnaire  (completed  by  course  developers  following 
extensive  interaction  with  the  devices),  structured  interviews  with  various 
knowledgeable  persons,  and  observations  provided  by  the  on-site  data  collector. 
Data  were  compiled  in  tables  and  reviewed  in  order  to  identify  similarities 
and  differences  in  opinion.  The  benefits  and  liabilities  of  each  prototype 
device  were  identified. 


Findings: 

Analysis  of  responses  indicated  that  the  AMTESS  concept  (i.e.,  generic, 
modular  maintenance  training  device)  is  viable,  particularly  to  train  stu¬ 
dents  where  use  of  operational  equipment  is  unsafe  or  may  result  in  equipment 
damage.  Comments  about  the  utility  of  specific  device  features  varied  with 
the  tasks  that  the  devices  trained.  In  general,  students,  instructors,  and 
course  developers  praised  a  variety  of  device  features  including  feedback 
provided  to  students,  the  ease  with  which  malfunctions  can  be  inserted  into 
the  devices,  and  the  quality  of  audio  and  visual  stimuli  presented  to  stu¬ 
dents.  The  low  reliability  and  durability  of  the  devices  detracted  from 
their  overall  effectiveness. 


Utilization  of  Findings; 

Results  of  this  and  other  reports  in  this  series  can  support  preparations 
of  guidelines  on  how  to  design  and  use  generic  maintenance  training  devices. 
Such  devices  can  increase  the  efficiency  and  effectiveness  of  Army  training. 
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ARMY  MAINTENANCE  TRAINING  AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS)  DEVICE 
EVALUATION:  VOLUME  III,  QUALITATIVE  ASSESSMENT  OF  TWO  PROTOTYPE  DEVICES 


INTRODUCTION 

The  Army  has  increased  its  emphasis  on  maintenance  training  simulation 
as  the  cost  and  complexity  of  weapons  systems  has  increased.  The  Army 
Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS)  program  is  an 
example  of  the  Amy's  emphasis  on  simulation.  Its  objective  is  to  provide 
cost-  and  training-effective  maintenance  simulators  (Dybas,  1981,  1983). 

Four  maintenance  trainer  concepts  were  developed  during  Phase  I  of  the  AMTESS 
program.  Grumman  Aerospace  Corporation  and  a  consortium  of  Seville  Research 
Corporation  and  Burtek,  Inc.  developed  breadboard  models  of  their  concepts 
during  Phase  II  of  the  program.  Each  breadboard  device  was  designed  so  that, 
by  attaching  unique  software  and  MOS-specific  three-dimensional  (3-0)  hardware 
to  the  core  computer  system,  the  device  could  be  used  to  teach  either  mechani¬ 
cal  or  electronic  maintenance. 

Both  the  Grumman  device  and  the  Seville/Burtek  device  are  composed  of 
four  major  components:  a  student  station,  an  instructor  station,  and  two  3-0 
modules.  The  Seville/Burtek  device  presented  curricula  to  students  via  CRT 
and  a  slide  projector.  Students  entered  troubleshooting  decisions  into  the 
device  by  manipulating  a  thumbwheel  and  pushbuttons  on  the  student  station. 
Students  trained  on  the  device  at  the  U.S.  Army  Ordnance  Center  and  School  at 
Aberdeen  Proving  Ground,  Maryland  (APG)  learned  to  identify  and  remove/replace 
faulty  components  (starter  motor,  oil  pump,  etc.)  on  a  3-0  module  that 
emulated  a  Cummins  NHC-250  diesel  engine.  Students  trained  on  the  device  at 
the  U.S.  Army  Air  Defense  School  at  Fort  Bliss,  Texas  were  trained  to  identify 
faulty  components  (modulation  circuits,  noise  degeneration  circuits,  etc.)  on 
a  3-0  module  that  emulated  an  Improved  Hawk  High  Power  Illuminator  Radar 
transmitter. 

The  Grumman  device  presented  curricula  to  students  with  an  interactive 
video  disc  system.  Students  entered  troubleshooting  decisions  by  pressing  a 
touch  panel  on  a  CRT.  Students  trained  with  this  device  at  APG  learned  to 
identify  faulty  components  on  a  3-0  module  that  emulated  the  starting  and 
charging  system  of  an  M110A2  self-propelled  howitzer.  At  Fort  Bliss,  students 
learned  to  identify  faulty  components  (modulation  circuits,  noise  degeneration 
circuits,  etc.)  on  a  3-0  module  that  emulated  the  Improved  Hawk  High  Power 
Illuminator  Radar  transmitter. 

The  results  of  a  1982  and  1983  evaluation  of  these  devices  are  presented 
in  a  three-volume  series  of  documents.  Volume  I  is  an  Executive  Overview. 
Volume  II  summarizes  an  evaluation  of  the  devices'  capability  to  provide 
transfer  of  training  to  operational  equipment.  (Students  were  trained  using 
either  conventional  methods  or  using  one  of  the  two  versions  of  the  AMTESS 
simulator  and  were  then  tested  on  ability  to  perform  actual  maintenance  tasks 
on  operational  equipment.)  Volume  III  (this  report)  summarizes  user  opinions 
about  the  usefulness  and  effectiveness  of  the  devices.  Other  reports  also 
address  the  AMTESS  program.  Front-end  analysis  procedures  were  discussed  by 
Woelfel,  Duffy,  Unger,  Swezey,  Hays,  and  Mirabella  (1984)  and  specific  device 
features  were  addressed  by  Criswell,  Unger,  Swezey,  and  Hays  (1983). 


The  AMTESS  devices  were  evaluated  in  six  experiments  at  two  locations 
(APG  and  Fort  Bliss)  with  soldiers  from  ^ight  Military  Occupational  Speciality 
areas  (MOSs).  Volume  II  explains  the  need  for  six  experiments.  Experiments 
1  and  2  addressed  the  Seville/Burtek  simulator  at  APG  while  Experitrients  3  and 
4  addressed  the  Grumman  simulator  at  APG.  Experiment  5  addressed  the  Seville/ 
Burtek  simulator  at  Fort  Bliss  while  Experiment  6  addressed  the  Grumman  simu¬ 
lator  at  Fort  Bliss  (see  Table  1). 

This  volume  descnoes  the  device  users'  opinions  which  were  collected 
as  part  of  the  transfer-of-training  evaluation  (Volume  II).  The  studies  in 
this  volume  will  be  refered  to  as  experiments  in  order  to  maintain  consistency 
with  Volume  II.  Table  1  shows  an  overview  of  the  evaluation  which  used  the 
following  data  collection  instruments: 

0  Initial  instructor  questionnaire.  These  forms  were  filled 
out  by  instructors  aifter  they  were  trained  to  use  the  simu¬ 
lators  but  prior  to  using  the  devices  to  train  students. 

0  Instructor  questionnaire.  Instructors  provided  detailed 
information  about  the  simulators  after  they  had  used  the 
devices  to  train  students. 

0  Course  developer  questionnaire.  Course  developers 
responded  to  questions  about  the  lessons  presented  on 
the  simulators  after  they  had  extensive  experience  with 
the  devices. 

0  Trainee  questionnaire.  After  students  completed  training 
with  the  simulator  and  testing  on  operational  equipment, 
they  responded  to  a  series  of  questions  concerning  their 
opinions  about  the  simulator. 

0  Structured  interview.  Various  individuals  who  were 
knowledgeable  about  the  simulators  responded  to  a  series 
of  13  questions  about  the  devices. 

0  Data  collector  observations.  During  the  transfer-of- 
training  evaluation,  data  collectors  recorded  various 
types  of  information  about  specific  device  features 
and  lessons.  They  also  assessed  the  reliability  of  the 
devices  and  various  aspects  of  trainee-simulator  interactions. 

This  volume  is  divided  into  three  major  sections.  Following  this  intro¬ 
duction,  the  second  section  presents  and  discusses  data  collected  in  each  of 
the  six  experiments  shown  in  Table  1.  The  final  section  provides  an  overall 
discussion  of  the  results  obtained  from  these  experiments. 


Table  1 


AMTESS  Field  Evaluation  Studies  Qualitative  Data 


1 

2 

Experiment 

3  4 

5 

6 

DEVICE 

S/B^ 

S/B 

G*' 

G 

S/B 

G 

MOS 

63830 

63W10 

63H30 

63D30 

24C10 

24E,  G,  R 

LOCATION 

APG 

APG 

APG 

APG 

BLISS 

BLISS 

SAMPLE  SIZE: 

Initial  Instructor 
Questionnaire 

2 

3 

5 

11 

2 

Instructor 

Questionnaire 

1 

1 

6 

1 

- 

Course  Developer 
Questionnaire 

2 

- 

3 

- 

- 

1 

Trainee 

Questionnaire 

20 

67 

15 

15 

10 

10 

Structured 

Interviews 

4 

1 

5 

- 

2 

5 

OTHER  DATA 

DCO^ 

- 

DCO 

- 

DCO 

DCO 

^Seville/Burtek 

^Grumman 

^Data  Collector  Observations 
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RESULTS  OF  EXPERIMENTS 


Experiment  1 

Forty  (40)  students  from  the  63B30  MOS  (Organizational  Maintenance 
Supervisor)  were  trained  to  perform  four  tasks  on  a  Cummins  NHC-250  diesel 
engine.  The  tasks  included  adjustment  of  alternator  belts,  removal /replacement 
of  the  starter  motor,  troubleshooting  an  oil  pump  failure,  and  inspection  of 
the  electrical  system.  Students  were  trained  with  either  conventional  methods 
(20  students)  or  the  Seville/Burtek  simulator  (20  students).  They  were  then 
tested  on  their  ability  to  perform  these  tasks  on  operational  equipment.  Data 
consist  of  responses  to  questionnaires  and  structured  interviews,  as  well  as 
observations  made  by  the  on-site  data  collector  during  the  course  of  the 
experiment. 

Method 

Subjects.  Characteristics  of  subjects  who  provided  data  are  presented 
below: 

0  Initial  instructor  questionnaire.  Two  Noncommissioned  . 

Officers  (NCOs)  who  were  instructors  for  the  63B30  MOS 
completed  the  questionnaire. 

0  Instructor  questionnaire.  One  of  the  instructors 
described  above  completed  this  questionnaire. 

0  Course  developer  questionnaire.  Two  civilians  responsible 
for  creating  and  modifying  curricula  for  the  63B30  MOS 
completed  this  questionnaire. 

0  Trainee  questionnaire.  Twenty  (20)  MOS  63B30  students 
who  had  received  training  with  the  Seville/Burtek 
simulator  completed  this  questionnaire  after  they  had 
been  tested  on  their  ability  to  perform  troubleshooting 
tasks  on  the  operational  equipment. 

0  Structured  interview.  Interviews  were  conducted  with  the 
two  instructors  described  above,  one  of  the  course  developers 
described  above,  and  the  data  collector  involved  in  the 
transfer-of-training  study  (Volume  II). 

Materials.  Four  questionnaires  and  one  structured  interview  form  (see 
Appendix  A)  were  used  to  collect  qualitative  data  about  the  Seville/Burtek 
device.  The  Initial  Instructor  Questionnaire  assessed  instructor  opinion 
about  the  simulator  after  instructors  had  been  trained  to  use  the  simulator, 
but  before  they  had  used  the  device  to  train  students.  Respondents  were 
asked  to  rate  (on  a  5-point  scale  ranging  from  "very  easy"  to  "very  hard") 
the  difficulty  involved  in  learning  to  use  the  simulator.  They  were  also 
asked  to  rate  (on  a  5-point  scale  ranging  from  "like  it  very  much"  to  "dislike 
it  very  much")  the  extent  to  which  they  liked  or  disliked  the  simulator.  Two 
other  questions  asked  respondents  to  comment  on  their  ratings  while  a  final 
question  asked  respondents  to  make  other  comments  they  felt  were  appropriate. 
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The  Instructor  Questionnaire  assessed  opinion  about  the  Seville/Burtek 
device  after  they  had  trained  students  on  the  simulator.  Respondents  were 
asked  to  rate  (on  a  5-point  scale  ranging  from  “very  well"  to  "very  poorly") 
how  well  the  device  trained  students  to  perform  tasks  addressed  by  the  lessons. 
[See  Table  2  for  a  description  of  the  22  tasks  (numbered  4-13  and  17-28)  that 
were  rated.]  They  were  also  asked  to  explain  why  they  made  this  rating  and 
to  comment  on  specific  device  features  involved  in  each  lesson.  A  final 
question  asked  them  to  make  other  comments  they  felt  were  appropriate. 

The  Course  Developer  Questionnaire  assessed  opinions  concerning  the 
appropriateness  of  the  curriculum  which  supported  the  Seville/Burtek  device. 

For  each  of  21  lessons  in  the  curriculum  that  were  evaluated,  the  respon¬ 
dents  noted  if  the  task  addressed  by  the  lesson  was  critical  and  if  the 
lesson  was  currently  taught  with  conventional  methods  (using  operational 
equipment).  Course  developers  were  asked  to  rate  (on  a  5-point  scale  ranging 
from  "very  difficult"  to  "very  easy")  the  difficulty  of  performing  each  task 
involved  in  the  lessons.  They  were  also  asked  to  rate  (on  a  5-point  scale 
ranging  from  "novice"  to  "expert")  the  skill  level  at  which  trainees  would 
perform  each  task  after  the  trainees  had  completed  training.  A  final  ques¬ 
tion  asked  them  to  make  other  comments  they  felt  were  relevant. 

The  Traine'^  Questionnaire  assessed  student  opinion  about  the  device 
after  students  had  completed  training  and  had  been  tested  on  their  ability  to 
perform  troubleshooting  tasks  on  operational  equipment.  Trainees  were  asked 
to  rate  (on  a  5-point  scale  ranging  from  "like  it  very  much"  to  "dislike  it 
very  much")  how  they  felt  about  the  simulator.  Subsequent  questions  asked 
them  to  explain  why  they  made  this  rating  and  to  comment  on  specific  device 
features  they  liked  or  disliked.  A  final  question  asked  students  to  make 
other  comments  they  felt  were  relevant. 

The  Structured  Interview  (13  questions)  addressed  a  variety  of  topics 
about  the  simulator,  its  features,  and  the  lessons  that  support  the  simulator. 
After  respondents  answered  the  questions,  they  were  asked  to  rate  (on  a  7- 
point  scale  ranging  from  "none"  to  "greatest")  the  simulator  features  (or 
lessons)  for  their  training  value  in  the  current  setting.  They  were  also 
asked  to  rate  (on. a  7-point  scale  ranging  from  "none"  to  "greatest")  the 
simulator  features  (or  lessons)  for  their  potential  training  value. 

Procedure.  Prior  to  filling  out  the  questionnaires  or  participating  in 
the  structured  interviews,  subjects  were  briefed  on  the  purpose  of  the  ques¬ 
tionnaires  (or  interview)  and  the  types  of  questions  they  would  be  asked. 

Initial  Instructor  Questionnaires  were  completed  in  a  group  setting 
after  instructors  had  completed  a  training  course  on  simulator  operation. 

One  instructor  and  one  MOS  63B30  student  worked  together  to  complete  22 
of  the  28  lessons  presented  by  the  simulator.  Responses  judged  to  be  redun¬ 
dant  were  combined  with  responses  to  other  lessons  in  order  to  save  time. 

The  instructor  filled  out  an  instructor  questionnaire  upon  completing  each  of 
the  22  lessons  in  which  he  participated. 

Course  Developer  Questionnaires  were  completed  as  two  course  developers 
and  one  MOS  63B30  student  worked  together  to  complete  21  of  the  22  lessons 


Seville/Burtek  Tasks  Rated  by  Instructors  and  Course  Developers  (Experiment  1) 


Task  Number  Description 

4  Remove  and  replace  oil  pump 

5  Perform  an  oil  pressure  test  using  STE/ICE 

6  Remove  and  replace  thermostat 

7  Remove  and  replace  water  pump 

8  Remove  and  replace  alternator 

9  Perform  a  DC  voltage  test  using  STE/ICE 

10  Remove  and  replace  starter  motor 

11  Perform  a  DC  current  test  using  STE/ICE 

12  Remove  and  replace  fuel  pump 

13  Perform  a  resistance  test  using  STE/ICE 

17  Troubleshoot  and  repair  engine  (oil  pump  failure-low  pressure) 

18  Troubleshoot  and  repair  the  cooling  system  (thermostat 

failure-low  temperature) 

19  Troubleshoot  and  repair  the  cooling  system  (water  pump 

failure-high  temperature) 

20  Troubleshoot  and  repair  fuel  system  (fuel  pump  failure- 

engine  stalls) 

21  Troubleshoot  and  repair  electrical  system  (starter  motor 

failure-slow  start) 

22  Troubleshoot  and  repair  electrical  system  (alternator 

failure-high  charge) 

23  Troubleshoot  and  repair  electrical  system  (loose  alternator 

belt-low  charge) 

24  Troubleshoot  and  repair  electrical  system  (alternator 

failure-BG  point  low) 

25  Troubleshoot  and  repair  electrical  system  (fuel  pump 

failure-hard  start) 

26  Troubleshoot  and  repair  electrical  system  (battery  switch 

failure) 

27  Troubleshoot  and  repair  electrical  system  (front  harness 

failure) 

28  Troubleshoot  and  repair  NHC-250  Diesel  (protective  control 

box  failure) 


that  were  completed  by  the  instructor  described  above.  Upon  completion  of 
each  lesson,  the  course  developers  independently  completed  a  questionnaire. 

Twenty  (20)  MOS  63B30  students  completed  Trainee  Questionnaires  after 
they  had  completed  training  and  they  had  been  tested  on  their  ability  to 
perform  troubleshooting  tasks  on  the  operational  equipment.  Trainees  com¬ 
pleted  their  questionnaires  independently. 

Two  instructors,  one  course  developer,  and  one  data  collector  completed 
the  Structured  Interview  after  all  transfer-of-training  data  had  been 
collected.  Interviews  were  conducted  on  an  individual  basis.  One  interview 
was  self-administered.  For  the  remaining  three  interviewees,  the  interview 
protocol  was  as  follows: 

0  The  interviewer  asked  the  interviewee  a  question. 

0  The  interviewer  prompted  the  interviewee  for  specific 
comments  about  the  simulator  if  the  interviewee  was 
initially  unresponsive. 

0  The  interviewer  asked  the  interviewee  to  rate  simulator 
features  (for  questions  1,  3,  4,  8,  9,  10,  11,  and  13). 

0  The  interviewer  reviewed  comments  with  the  interviewee 
to  ensure  correctness  of  recorded  responses. 

Results^ 

Several  of  the  questionnaires  asked  respondents  to  comment  on  specific 
device  features.  In  order  to  clarify  these  comments,  brief  descriptions  of 
the  major  features  of  the  Seville/Burtek  device  are  presented  in  Table  3. 

Initial  Instructor  Questionnaire.  Results  of  the  initial  instructor 
questionnaire  are  summarized  below. 

QUESTION  1:  How  easy  or  hard  is  it  to  learn  to  use  the  simulator? 

RATINGS:  1  *  very  hard,  5  ■  very  easy 

N  =  2 


Response 


Number  of  Responses 


VERY  EASY  2 
FAIRLY  EASY  0 
AVERAGE  0 
FAIRLY  HARD  0 
VERY  HARD  0 


TOTAL  7 


^Data  presented  in  this  section  for  each  of  the  six  experiments  reported  are 
available  in  raw  form  from  the  U.S.  Am\y  Research  Institute. 
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Table  3 


Description  of  Seville/Burtek  Device  Features 


Feature 

Description 

Remove/replace 
capabi 1 i ty 

Components  of  the  3-D  module  could  be  removed  and 
replaced  by  students. 

Student  CRT 
unit 

Text  presented  on  this  CRT  directed  student  actions  and 
provided  feedback.  ^ 

Students  turned  a  thumbwheel  and  pushed  buttons  to  enter 
their  decisions  (i.e.,  inspect  hoses)  into  the  train¬ 
ing  device. 

Student 

responder  unit 

Instructor  CRT 
unit 

Instructors  were  able  to  monitor  student  actions  on 
this  CRT  unit. 

Hardcopy 
pri ntout 

A  record  of  student  performance  could  be  printed  as  the 
student  performed  the  lesson  or  after  the  lesson  was 
completed. 

Lesson 

arrangement 

The  sequence  in  which  students  participated  in  lessons 
could  be  controlled  by  the  instructor. 

Editing  system 

Text,  graphics,  and  all  procedures  involved  in  a  lesson 
could  be  modih'ed  by  instructors. 

Malfunction 

insertion 

Numerous  faults  could  be  inserted  into  the  simulator  by 
manipulating  two  controls  on  the  instructor  station. 

Random  malfunc¬ 
tion  insertion 

This  feature  randomly  selects  a  malfunction  and  inserts 
it  into  the  simulator. 

Slide  projector 
unit 

Photographs,  diagrams,  and  other  visual  aids  were  pre¬ 
sented  via  slide  projector. 

Sound  effects 

Various  equipment  sounds  were  simulated. 

Note:  The  3-D  module  of  the  Seville/Burtek  device  for  the  63B30  and  63W10 
MOSs  (Experiments  1  and  2)  is  a  full-scale  reproduction  of  a  Cummins 
NHC-250  diesel  engine.  The  3-D  module  is  composed  of  engine  components 
(i.e.,  starter  motor,  batteries,  etc.),  controls,  displays,  and  test 
equipment. 

The  3-D  module  of  the  Seville/Burtek  device  used  by  the  24C10  MOS 
students  (Experiment  5)  is  a  full-scale  reproduction  of  an  Improved 
Hawk  High  Power  Illuminator  Radar  transmitter  unit.  The  3-D  module 
is  composed  of  a  cabinet,  interior  components  (i.e.,  cables,  power 
supplies,  etc.),  controls,  displays,  and  test  equipment. 
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When  asked  to  rate  the  difficulty  involved  in  learning  to  use  the 
Seville/Burtek  device,  two  instructors  indicated  that  it  was  very  easy  to 
learn  to  use  the  device. 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Instructors  indicated  that  it  was  very  easy  to  learn  to  use  the  simulator 
because  the  manufacturer  had  provided  detailed  instructions  for  operating  the 
device. 

QUESTION  3:  How  do  you  feel  about  the  simulator? 

RATINGS:  1  =  dislike  it  very  much,  5  =  like  it  very  much 

N  =  2 


I 


Response  Number  of  Responses 

LIKE  IT  VERY  MUCH  2 

LIKE  IT  SOMEWHAT  0 

NO  STRONG  FEELINGS  ABOUT  IT  0 

DISLIKE  IT  SOMEWHAT  0 

DISLIKE  IT  VERY  MUCH  0 

TOTAL  I 

QUESTION  4:  Why  did  you  answer  Question  3  as  you  did? 

The  instructors  rated  the  device  highly  because  they  felt  it  would  be  a 
useful  training  device  and  that  it  was  easier  to  induce  malfunctions  in  the 
simulator  than  in  operational  equipment. 


QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

Additional  comments  made  by  the  instructors  indicated  that  they  felt  the 
device  would  be  useful  for  Skill  Qualification  Tests  (SQTs).  They  also  stated 
that  it  would  be  useful  to  place  the  device  at  the  battalion  level. 

Instructor  Questionnaire.  Data  reported  for  the  instructor  questionnaire 
are  based  on  the  responses  of  one  instructor. 

QUESTION  1:  How  well  does  the  simulator  train  this  task? 

RATINGS:  1  =  very  poorly,  5  =  very  well 

N  =  1 


r 

Response 

Number  of  Responses 

VERY  WELL 

7 

1 

MODERATELY  WELL 

13 

» * 

• 

FAIRLY 

2 

POORLY 

0 

VERY  POORLY 

0 

TOTAL  22 

i' 

1 

One  questionnaire  was  complete. 

.y.'  each  of  22  tasks 

simulator  curriculum.  When  asked  '.j  rate  how  well  the  simulator  trains  each 
of  the  22  tasks,  the  instructor  idicated  that  most  tasks  were  trained 
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moderately  well.  The  mean  rating  for  all  tasks  was  4.23  (standard  devia¬ 
tion  =  6.1).  Raw  data  are  presented  in  Table  B-1. 

QUESTION  2;  Why  did  you  answer  Question  1  as  you  did? 

Instructor  comments  indicated  that  "fair"  ratings  were  given  to  tasks 
that  were  poorly  documented;  that  is,  directions  provided  to  students  on  the 
student  CRT  or  in  the  student  guidebook  (provided  by  Seville/Burtek  to  help 
students  complete  lessons)  were  unclear  and  did  not  match  directions  provided 
in  TMs.  The  instructor  did  not  indicate  why  certain  tasks  were  given  a 
rating  of  "very  well"  and  other  tasks  were  given  a  rating  of  "moderately  well." 

QUESTION  3:  What  specific  features  of  the  simulator  (involved  in  this 
task)  do  you  like  or  dislike? 

Specific  features  of  the  simulator  that  appealed  to  the  instructor  were: 

1)  high  fidelity  of  the  alternator  component,  2)  ease  of  inserting  malfunc¬ 
tions  into  the  simulator,  and  3)  feedback  provided  to  the  student.  The 
instructor  disliked  the  low  fidelity  representation  of  the  starter  motor 
remove/replace  task,  and  the  sensitivity  of  the  3-D  module  during  this  task. 

QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

T^a  instructor  disliked  the  low  physical  fidelity  of  components  involved 
in  the  starter  motor  remove/replace  task  because  it  made  the  task  too  simple 
to  perform.  He  stated  that  obstacles  to  removing  this  component  (propeller 
shaft  and  frame)  are  not  represented  in  the  simulator.  He  also  felt  that  the 
sensitivity  of  the  3-D  module  resulted  in  a  student  performance  record  that 
included  an  inflated  number  of  errors. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

The  instructor  indicated  that  the  student  guidebook  provided  little  use¬ 
ful  information  for  tasks  6  and  9.  He  also  stated  that  he  liked  the  way  tasks 
27  and  28  were  presented  on  the  simulator. 

Course  Developer  Questionnaire.  Two  course  developers  completed  a  ques- 
tionnaire  for  each  of  21  tasks  included  in  the  simulator  curriculum. 

QUESTION  1:  Is  this  a  critical  task  which  needs  to  be  trained? 

RESPONSES:  Yes,  No. 

N  =  2 

Response  Number  of  Responses 

YES  8 

NO  34 

TOTAL  ?2 

One  course  developer  indicated  that  all  of  the  rated  tasks  were  critical, 
while  a  second  course  developer  indicated  that  only  13  of  the  21  rated  tasks 
were  critical.  Respondents  made  no  comments  about  their  decisions  concerning 
task  criticality.  Raw  data  are  presented  in  Table  3-2. 


QUESTION  2:  Is  this  task  currently  trained  on  operational  equipment? 
RESPONSES:  Yes,  No 
N  =  2 


Course  developers  agreed  that  nearly  all  of  the  tasks  included  in 
the  simulator  curriculum  were  currently  included  in  conventional  training. 
Raw  data  are  presented  in  Table  B-3. 

QUESTION  3:  How  difficult  is  it  to  perform  this  task? 

MTINGS:  1  *  very  easy,  5  »  very  difficult 
N  =  2 


Response 


Number  of  Responses 


VERY  DIFFICULT  0 
SOMEWHAT  DIFFICULT  25 
ABOUT  AVERAGE  IN  DIFFICULTY  9 
SOMEWHAT  EASY  8 
VERY  EASY  0 


TOTAL  ^ 


When  asked  to  rate  the  difficulty  of  performing  tasks  included  in  the 
simulator  curriculum,  the  course  developers  concurred  that,  on  average,  the 
tasks  were  somewhat  above  average  in  difficulty.  The  mean  rating  for  all 
tasks  was  3.4  (standard  deviation  «  .8).  Raw  data  are  presented  in  Table 
B-4. 


QUESTION  4:  At  what  skill  level  should  a  trainee  perform  this  task  at 
the  end  of  training? 

RATINGS:  1  ®  novice,  5  =  expert 
N  =  2 


Response 


Number  of  Responses 


EXPERT  0 
LESS  THAN  EXPERT  18 
APPRENTICE  24 
BETTER  THAN  NOVICE  0 
NOVICE  0 


TOTAL  ^ 


When  asked  to  rate  the  skill  level  at  which  trainees  should  perform  a 
task  upon  completion  of  training,  course  developers  indicated  that,  on 
average,  trainees  would  perform  somewhat  better  than  apprentice  level.  The 
mean  rating  for  all  tasks  was  3.43  (standard  deviation  =  .5).  Raw  data  are 
presented  in  Table  B-5. 


QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 
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The  course  developers  made  several  additional  comments.  They  both  noted 
that  the  simulator  curriculum  would  have  been  better  if  it  had  included  set¬ 
up  and  check-out  procedures  for  the  Simplified  Test  Equipment/ Internal 
Combustion  Engine  (STE/ICE)  kit.  The  course  developers  also  agreed  that  it 
may  be  more  appropriate  to  provide  training  for  remove/replace  tasks  using 
operational  equipment  since  physical  fidelity  is  more  important  for  remove/ 
replace  tasks  than  it  is  for  troubleshooting  tasks.  One  course  developer 
made  several  comments  identifying  erroneous  diagrams,  procedures,  etc. 
associated  with  the  simulator  curriculum.  Another  course  developer  noted 
the  ease  of  inserting  malfunctions  into  the  simulator  and  the  decreased 
risk  of  student  injury  associated  with  the  simulator. 

Trainee  Questionnaire.  Trainees  completed  questionnaires  after  they  had 
been  trained  with  the  simulator  and  had  been  tested  on  their  ability  to  per¬ 
form  four  troubleshooting  tasks  on  operational  equipment. 

QUESTION  1:  How  do  you  feel  about  the  simulator? 

RATINGS:  1  =  dislike  it  very  much,  5  =  like  it  very  much 

N  =  20 


Response  Number  of  Responses 

LIKE  IT  VERY  MUCH  16 

LIKE  IT  SOMEWHAT  4 

NO  STRONG  FEELINGS  ABOUT  IT  0 

DISLIKE  IT  SOMEWHAT  0 

DISLIKE  IT  VERY  MUCH  0 

TOTAL  US 

Trainees  rated  the  simulator  highly.  The  mean  rating  for  all  students 
was  4.8  (standard  deviation  =  .41). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Many  students  indicated  that  they  rated  the  simulator  highly  because 
they  felt  they  were  well -trained. 

QUESTION  3:  What  specific  features  of  the  simulator  do  you  like  or  dis¬ 
like? 


Two  students  stated  that  they  liked  all  device  features.  Specific 
features  that  students  liked  include  feedback,  hands-on  capability,  simplicity 
of  operation,  and  the  hardcopy  printout  of  student  performance.  Several 
students  disliked  the  fact  that  the  simulator  was  so  sensitive  to  vibration 
during  remove/ re pi  ace  tasks  that  it  recorded  minor  movements  as  deviations 
from  normal  procedures. 

QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

There  were  no  relevant  responses  to  this  question. 
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QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

Many  students  commented  that  they  enjoyed  working  with  the  simulator. 

Structured  Interview.  Responses  to  the  13  questions  posed  during  the 
structured  interview  are  summarized  below. 

QUESTION  1:  What  were  the  instructional  features  of  the  simulator  that 
were  applicable  to  the  school's  training  course? 

Four  interviewees  stated  that  the  3-D  module  was  applicable  to  the 
school's  training  course.  These  interviewees  criticized  the  3-D  module, 
however,  because  it  lacked  the  required  degree  of  physical  or  functional 
fidelity  for  components  such  as  the  STE/ICE  kit,  oil  dipstick,  propeller 
shaft,  and  radiator  fan. 

Opinions  were  mixed  concerning  the  applicability  of  feedback  provided 
on  the  student  CRT  unit.  One  interviewee  felt  that  the  feedback  helped 
students  complete  lessons.  Another  interviewee,  however,  felt  that  the  feed¬ 
back  confused  students  at  times.  This  confusion  resulted  when  the  students 
performed  tasks  in  an  order  that  was  considered  acceptable  by  school  per¬ 
sonnel,  but  the  simulator  provided  feedback  that  indicated  that  the  order 
was  inappropriate. 

Three  interviewees  stated  that  the  hardcopy  printout  of  student  perform¬ 
ance  is  applicable  to  the  school's  training  course.  One  interviewee,  however, 
felt  that  the  printout  was  too  detailed.  He  noted  that  detailed  performance 
records  were  unnecessary  at  AP6. 

Three  interviewee?  stated  that  the  flexible  arrangement  of  lessons  was 
useful.  0''e  interviewee  indicated  that  a  simulator  that  did  not  incorporate 
this  feature  would  be  of  limited  value. 

All  interviewees  stated  that  the  abil.ty  to  inse'^t  malfunctions  into  the 
simulator  was  highly  useful.  One  interviewee  noted  that  it  was  very  easy  to 
insert  several  malfunctions  at  a  time  into  the  simci.ztor. 

While  interviewees  thought  that  the  remove/replace  feature  was  applicable, 
they  noted  several  deficiencies:  1)  components  are  not  durable  and  cannot 
withstand  normal  wear  and  tear,  and  2)  since  components  are  easily  accessed, 
much  of  the  difficulty  associated  with  remove/repl ace  tasks  is  eliminated. 

Opinions  concerning  the  simulator's  editing  feature  varied  considerably. 
Two  interviews  thought  it  would  be  highly  useful  to  be  able  to  change  lesson 
content  and  arrangement.  Others,  however,  stated  that  the  editing  system 
was  of  little  value. 

Three  individuals  indicated  that  the  random  malfunction  selection  feature 
was  moderately  useful  while  one  stated  that  it  was  of  little  value.  One 
individual  noted  that  this  feature  may  be  more  appropriate  for  SQT  than  for 
training  63B30  students.  Another  interviewee  noted  that  selection  of  mal¬ 
functions  did  not  seem  to  be  random.  Rather,  the  same  subset  of  malfunctions 
were  consistently  selected  by  the  simulator. 
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Interviewee  assessment  of  the  slide  projector  unit  ranged  from  "good"  to 
"poor."  The  interviewee  who  liked  this  feature  praised  it  for  presenting 
illustrations  that  could  not  normally  be  shown  during  training  (i.e.,  leaking 
oil  lines).  Another  interviewee,  however,  felt  that  the  projector  unit  was 
of  little  value  because  most  of  the  material  it  presented  came  from  existing 
technical  manuals  (TMs). 

Two  interviewees  stated  that  they  liked  the  self-paced  lesson  format. 
They  also  noted  that  a  self-paced  format  did  not  remove  the  requirement  for 
an  instructor's  presence  during  training. 


Ratings  made  by  the  interviewees 

are  presented 

below: 

RATINGS 

(1  *  None,  7  »  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

4 

4 

.82 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

4 

5.5 

.58 

QUESTION  2:  What  were  the  instructional  features  of  the  simulator  that 
were  not  applicable  to  the  school's  training  course? 

Interviewees  indicated  that  few  of  the  instructional  features  were  not 
applicable.  Rather,  they  described  some  of  the  less  desirable  aspects  of 
the  applicable  features  described  in  Question  1.  Features  that  were 
described  as  inapplicable  include  the  hardcopy  printout  of  student  perform¬ 
ance,  the  editing  system,  random  malfunction  selection,  and  the  slide  pro¬ 
jector  unit.  Each  of  these  features  was  thought  to  be  inapplicable  by  one 
of  the  four  interviewees. 

QUESTION  3:  Which  features  of  the  lesson  presentation  helped  make  the 
simulator  lessons  interesting  to  the  students? 

The  interviewees  generally  agreed  that  students  enjoyed  troubleshooting 
the  3-D  module  because  it  faithfully  reproduced  symptoms  of  a  malfunctioning 
engine.  Interviewees  also  felt  that  the  simulator's  sound  effects  played 
an  important  part  in  maintaining  student  interest  in  the  lessons.  Addition¬ 
ally,  most  of  the  interviewees  stated  that  the  materials  presented  on  the 
slide  projector  unit  held  students'  attention. 


Ratings  by  the  interviewees  are  presented  below; 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

4 

4.5 

1.29 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

4 

5.25 

.5 

QUESTION  4:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  particularly  effective? 

One  interviewee  (an  instructor)  indicated  that  the  lessons  involving 
troubleshooting  (identifying  faulty  components)  were  more  effective  than 
lessons  involving  removal  and  replacement  of  components.  He  stated  that  the 
effectiveness  of  certain  remove  and  replace  tasks  (i.e.,  starter  motor)  was 
reduced  because  the  physical  fidelity  of  the  simulator  was  somewhat  less  than 
optimal.  He  also  noted  that  lessons  dealing  with  components  that  were 
represented  in  the  simulator  (oil  pump)  were  more  effective  than  lessons 
dealing  with  components  that  were  not  represented  in  the  simulator  (i.e., 
the  driver's  compartment). 

Ratings  made  by  the  interviewees  are  presented  below: 


RATING 

MEAN 

STANDARD 

(1  *  None,  7  *  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  lessons  as  a  whole. 

4 

4.75 

.5 

Rate  the  potential  training 
value  of  these  lessons  as  a 
whole. 

4 

4.5 

.58 

QUESTION  5:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  ineffective? 

Several  interviewees  stated  that  the  lessons  dealing  with  the  STE/ICE 
kit  were  ineffective.  They  stated  that  lessons  involving  STE/ICE  did  not 
include  the  "set-up  and  check-out"  procedures  that  are  normally  presented 
during  conventional  training. 

QUESTION  6:  Were  there  any  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  being  presented  by  the  simulator  was  above  the  students? 

None  of  the  interviewees  felt  that  any  of  the  material  presented  to 
students  was  too  difficult  for  the  students  to  understand. 

QUESTION  7:  Were  there  any  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  being  presented  by  the  simulator  should  be  increased? 
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Interviewees  identified  three  areas  where  the  difficulty  level  of  the 
presented  material  should  be  increased.  First,  interviewees  stated  that  the 
materials  presented  on  the  slide  projector  unit  were  too  simplistic.  They 
felt  that  materials  presented  on  the  projector  would  be  more  useful  if  they 
illustrated  aspects  of  the  engine  that  are  not  normally  seen  on  operational 
equipment  (i.e.,  leaking  oil  lines).  Second,  the  interviewees  indicated  that 
certain  remove/ rep lace  tasks  were  presented  in  a  simplistic  manner.  One 
interviewee  noted  that  the  remove/ replace  starter  motor  lesson  was  too  easy 
for  students  because  the  3-D  module  provided  easy  access  to  this  component. 
Another  interviewee  noted  that  remove/ replace  tasks  on  the  simulator  were 
too  simplistic  because  these  lessons  did  not  require  students  to  apply  appro¬ 
priate  levels  of  torque  when  tightening  bolts.  The  third  area  mentioned  by 
interviewees  concerns  the  use  of  TMs.  Several  interviewees  stated  that  tasks 
were  simplified  because  simulator  lessons  indicated  precise  TM  page  and  para¬ 
graph  numbers.  They  felt  that  training  would  have  been  improved  if  the  stu¬ 
dents  had  been  trair.id  to  find  the  correct  TM  pages  for  themselves. 

QUESTION  8:  What  hardware  features  of  the  simulator  made  it  more 
effective  than  conventional  training? 

Interviewees  generally  agreed  on  the  features  of  the  simulator  that  made 
it  more  effective  than  conventional  training.  They  indicated  that  malfunctions 
could  be  inserted  into  the  simulator  much  more  easily  than  they  could  be 
inserted  into  operational  equipment.  One  interviewee  noted  that  certain  mal¬ 
functions  inserted  into  the  simulator  could  not  be  inserted  into  the  opera¬ 
tional  equipment  without  considerable  damage  to  the  equipment  (e.g.,  an  oil 
pump  failure). 

The  interviewees  also  agreed  that  the  low  voltages  present  in  the  simu¬ 
lator  significantly  reduced  the  possibility  that  students  would  injure  them¬ 
selves  during  training.  The  hardcopy  printout  was  another  feature  that  inter¬ 
viewees  mentioned.  One  interviewee  stated  that  it  was  effective  to  use  this 
printout  when  debriefing  students.  Consistent  presentation  of  material  to 
students  was  also  mentioned  as  a  feature  of  the  simulator  that  made  simulator 
training  more  effective  than  conventional  training. 

Ratings  made  by  the  interviewees  are  presented  below: 

RATINGS  MEAN  STANDARD 

(1  =  None,  7  =  Greatest)  N  RATING  DEVIATION 

Rate  the  training  value 
of  these  features  as  a 

whole.  4  4.25  .5 

Rate  tne  potential 
training  value  of  these 

features  as  a  whole.  4  4.75  .5 

QUESTION  9:  What  were  the  features  that  made  this  simulator  easy  for 
the  instructors  to  operate? 

Several  interviewees  stated  that  training  sessions  conducted  with  the 
simulator  were  simplified  because  of  the  easy  access  to  components  of  the  3-D 
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module.  They  noted  that  when  training  sessions  are  conducted  with  operational 
equipment,  components  may  not  be  clearly  seen  or  reached  by  students. 

The  interviewees  who  felt  that  the  editing  system  was  applicable  to  the 
school's  training  course  (Question  1)  felt  that  the  system  made  the  simulator 
easy  to  operate.  They  noted  that  lessons  could  be  easily  changed  with  this 
system.  The  instructor  CRT  and  instructor  panel  were  also  cited  by  the  inter¬ 
viewees  as  facilitating  operation  of  the  simulator. 


Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

4 

5.5 

.58 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

3 

5.3 

.58 

QUESTION  10:  What  were  the 

features 

that  made 

this  simulator  easy  for 

students  to  operate? 


The'e  were  very  few  responses  to  this  question.  Interviewees  had  many 
comments  on  the  features  that  made  the  simulator  difficult  for  the  students 
to  operate  (see  Question  12),  but  few  comments  on  features  that  facilitated 
student  performance.  When  responses  were  made,  the  interviewees  agreed  that 
students'  interaction  with  the  simulator  was  facilitated  because  the  module 
allowed  easy  access  to  components,  it  faithfully  represented  the  physical 
and  functional  characteristics  of  the  operational  equipment,  and  it  was  not 
a  safety  hazard. 

Interviewees  also  felt  that  the  student  guidebooks  helped  students  com¬ 
plete  lessons  successfully.  This  statement  contrasts  with  the  instructor 
who  felt  that  student  guidebooks  were  not  useful  for  tasks  six  and  nine  (see 
Question  5,  page  10). 

Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

FATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

4 

4.25 

.96 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

4 

5.25 

.5 

QUESTION  11:  What  aspects  of  the  device  would  be  appropriate  for  sub¬ 
stituting  for  actual  equipment? 
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The  interviewees  agreed  that  the  3-D  module  could  be  substituted  for 
operational  equipment.  Several  of  them  also  stated  that  the  question  was 
difficult  to  answer  since  they  felt  it  would  be  impossible  to  substitute 
specific  features  of  the  device  for  the  operational  equipment  without  sub¬ 
stituting  the  entire  device.  It  appeared  that  the  interviewees  then  inter¬ 
preted  the  question  to  read  "What  aspects  of  the  device  make  it  appropriate 
for  substituting  for  actual  equipment?".  Bearing  this  interpretation  in 
mind,  the  interviewees  indicated  that  the  ability  to  insert  a  variety  of 
malfunctions  into  the  simulator,  the  accessibility  of  simulator  components, 
and  the  decreased  probability  of  student  injury  on  the  simulator  made  the 
device  substitutable  for  actual  equipment. 

Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 


whole. 

4 

4.25 

.96 

Rate  the  potential 

training  value  of  these 

features  as  a  whole. 

4 

5.25 

.5 

QUESTION  12:  What  types  of  problems  did  the  students  have? 

One  of  the  interviewees  did  not  respond  to  the  question  as  he  did  not 
work  with  students.  The  remaining  interviewees  listed  a  variety  of  problems 
experienced  by  students. 

The  slide  projector  unit  was  seen  as  unreliable  by  several  interviewees 
One  interviewee  stated  that  numerous  delays  in  training  were  caused  by  this 
unit's  tendency  to  overheat  and  melt  slides.  The  interviewees  felt  that 
students  were  confused  by  the  variety  of  materials  that  required  their  atten 
tion.  One  interviewee  noted  that  students  had  to  attend  to  the  student  CRT, 
the  slide  projector  unit,  the  3-D  module,  the  student  responder  panel,  TMs, 
a  student  guidebook,  ar.u  a  list  of  potential  malfunctions.  Interviewees 
felt  that  the  student  responder  panel  was  difficult  for  students  to  under¬ 
stand.  They  indicated  that  students  spent  too  much  time  trying  to  enter 
their  responses  into  the  device.  Interviewees  indicated  that  during  lengthy 
troubleshooting  procedures,  little  feedback  was  provided  to  students.  Con¬ 
sequently,  students  did  not  know  if  they  were  following  procedures  correctly 
Another  interviewee  noted  that  when  feedback  was  present  it  provided  too 
little  information  to  be  of  value  to  students.  One  interviewee  noted  that 
some  sections  of  the  student  guidebook  did  not  follow  the  information  pre¬ 
sented  in  TMs  or  information  presented  on  the  student  CRT  and  the  slide  pro¬ 
jector  unit.  The  interviewee  went  on  to  say  that  when  students  were  con¬ 
fused,  the  simulator  did  not  afford  them  the  opportunity  to  repeat  a  step 
without  seeking  the  assistance  of  an  instructor.  As  was  noted  previously, 
the  interviewees  stated  that  students  experienced  difficulty  with  lessons 
involving  STE/ICE  because  these  lessons  did  not  include  procedures  required 
to  set  up  and  check  out  this  equipment. 


QUESTION  13:  How  would  you  employ  the  simulator  in  order  to  gain  maxi¬ 
mum  benefit  from  it? 

Several  interviewees  stated  that  the  highly  skilled  63B30  students  would 
not  benefit  from  the  remove/replace  lessons  as  much  as  students  who  were  less 
skilled.  They  stated  that  remove/replace  tasks  are  more  appropriate  for  "10" 
level  students  and  less  appropriate  for  "30"  level  students.  The  interviewees 
stated  that  the  "30"  level  students  should  only  use  the  simulator  for 
refresher  training  or  for  testing  purposes. 

Several  interviewees  stated  that  the  simulator  should  be  used  in  con¬ 
junction  with  operational  equipment  in  order  for  students  to  experience  the 
increased  difficulties  associated  with  removing  and  replacing  components  from 
the  operational  equipment. 

Ratings  made  by  the  interviewees  for  the  simulator  as  a  whole  are 
presented  below: 


RATINGS 

(1  =  None.  7  ■  Greatest) 

MEAN 

N  RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  the  simulator  as  a 
whole. 

4  4.75 

.5 

Rate  the  potential 
training  value  of  the 
simulator  as  a  whole. 

4  5.25 

.5 

Data  Collector  Observations.  During  the  transfer-of-training  experiment, 
the  data  collector  was  able  to  make  a  variety  of  observations  about  the 
operation  of  the  simulator.  These  observations  are  summarized  below: 

0  Electrical  sensors  on  the  3-D  engine  module  enabled  the 
computer  to  assess  the  student's  progress  through  a 
remove/ repl ace  lesson.  These  sensors  were  overly  sensitive, 
however,  and  produced  erroneous  data  on  the  student's 
performance  record.  Students  were  confused  by  such  errors 
on  their  performance  record. 

0  The  slide  projector  unit  frequently  overheated,  jammed, 
and  melted  slides.  Problems  with  this  unit  caused 
numerous  delays  in  the  training  program. 

0  Instructors  repeatedly  warned  students  that  nuts  and 
bolts  could  only  be  hand  tightened  because  the  device 
was  delicate.  Despite  this  warning,  students 
occasionally  stripped  threads  when  attempting  to 
remove  or  replace  a  component  on  the  3-D  module. 

0  In  general,  the  3-D  module  is  a  high  fidelity 

representation  of  a  diesel  engine.  Certain  components, 
however,  are  not  represented  in  the  simulator  and,  in 
some  cases,  the  absence  of  these  components  seriously 
affected  training.  For  example,  students  can 


remove/ replace  the  starter  motor  on  the  simulator 
much  more  easily  than  they  can  on  operational  equipment 
because  certain  obstacles  (the  propeller  shaft,  frame) 
are  not  represented  on  the  simulator.  Students  trained 
to  perform  this  task  using  the  simulator  experience 
difficulty  performing  the  task  on  operational  equipment. 

0  Sound  effects  present  during  training  sessions  helped 
to  maintain  students*  motivation. 

0  The  editing  system  is  very  flexible.  Instructors 
could  have  completely  reorganized  simulator  lessons 
if  they  had  desired  to  do  so. 

0  Students  find  the  student  responder  panel  difficult 
to  operate. 

0  Students  were  frequently  confused  by  the  considerable 
number  of  stimuli  to  which  they  were  required  to  attend. 
Students  attended  to  the  CRT,  slide  projector  unit,  3-D 
module,  responder  panel,  TMs,  and  student  guidebook. 

Discussion 


Although  the  quantity  of  data  reported  for  Experiment  1  is  considerable, 
analysis  of  this  data  is  .implified  since  many  of  the  coimjents  made  by 
different  individuals  in  response  to  different  questions  are  quite  similar. 

The  data  can  be  summarized  by  the  four  positive  statements  and  the  five 
negative  statements  about  the  simulator  that  are  discussed  below. 

Positive  Statements.  Instructors,  course  developers,  and  trainees  hold 
favorable  opinions  of  the  S'evi11e/6urtek  simulator.  Instructors  indicated 
that  they  had  little  difficulty  learning  to  operate  the  simulator.  They  also 
felt  that  students  trained  with  the  simulator  would  be  able  to  perform  quite 
well  on  operational  equipment.  Course  developers  indicated  that  the  tasks 
addressed  by  the  simulator  curriculum  were  critical  and  that  students  trained 
with  the  simulator  would  perform  well  on  operational  equipment.  Trainees 
indicated  that  they  enjoyed  working  with  the  simulator,  and  they  felt  well- 
prepared  to  perform  troubleshooting  tasks  on  operational  equipment. 

Ease  of  inserting  malfunctions  is  a  valuable  device  feature.  Instructors 
and  course  developers  appreciated  the  fact  that  a  variety  of  malfunctions 
could  be  inserted  into  the  simulator  with  minimal  effort.  Further,  they 
praised  the  device  for  its  ability  to  simulate  faults  that  could  not  be 
induced  in  operational  equipment. 

Performance  monitoring  is  a  valuable  device  feature.  Instructors  and 
trainees  indicated  that  training  was  facilitated  because  they  could  identify 
student  errors  as  they  were  being  committed.  Instructors  also  praised  the 
hardcopy  printout  of  student  performance  for  its  value  when  debriefing  students. 

The  simulator  is  safer  than  operational  equipment.  Both  instructors  and 
course  developers  noted  that  students  were  less  likely  to  injure  themselves 


on  the  simulator  than  on  operational  equipment  because  low  voltages  are 
present  in  the  simulator. 

Negative  Statements.  Students  were  confused  by  the  materials  to  which 
they  must  attend.  Instructors,  course  developers,  and  the  data  collector 
indicated  that  students  did  not  always  attend  to  the  correct  stimulus  during 
training  because  there  were  numerous  potential  sources  of  information. 

Physical  fidelity  of  the  3-0  module  is  too  low  for  certain  remove/replace 
tasks.  Instructors,  course  developers,  and  the  data  collector  felt  that  while 
the  easy  access  to  components  found  on  the  3-D  module  was  beneficial,  certain 
remove/ replace  tasks  were  too  simplistic  because  students  did  not  have  to 
maneuver  around  obstacles  that  are  present  on  operational  equipment. 

The  reliability  and  durability  of  the  simulator  should  be  improved. 

Many  individuals  noted  that  the  slide  projector  unit  failed  repeatedly  and 
that  components  of  the  3-D  module  could  not  withstand  with  rigors  of  normal 
use. 


Lessons  should  have  included  procedures  for  set  up  and  check  out  of  the 
STE/ICE  kit.  Although  several  lessons  contained  minor  procedural  errors, 
statements  made  by  instructors,  students,  course  developers,  and  the  data 
collector  indicated  that  the  exclusion  of  the  STE/ICE  set-up  and  check-out 
procedures  from  the  simulator  curriculum  significantly  decreased  the  training 
value  of  the  simulator. 

The  3-D  module  is  too  sensitive.  Instructors,  students,  and  the 
data  collector  indicated  that  minor  vibrations  that  normally  accompany 
remove/ replace  operations  are  recorded  as  errors. 


Experiment  2 


Forty-one  (41)  students  from  the  63W10  MOS  (Direct  Support  Vehicle 
Repairman)  were  trained  to  troubleshoot,  remove,  and  replace  an  oil  pump 
using  either  conventional  methods  (20  students)  or  the  Seville/Burtek  simula¬ 
tor  (21  students).  Students  were  then  tested  on  their  ability  to  perform 
this  task  on  operational  equipment.  Data  consist  of  responses  to  question¬ 
naires  and  structured  interviews,  as  well  as  various  observations  made  by  the 
on-site  data  collector  during  the  course  of  the  experiment. 

Method 


Subjects.  Characteristics  of  subjects  who  provided  data  are  as  follow: 

0  Initial  instructor  questionnaire.  Three  NCOs  who  were 
instructors  for  the  63W10  m6S  completed  the  questionnaire. 

0  Instructor  questionnaire.  This  questionnaire  was  com- 
pleted  by  one  of  three  instructors  described  above. 

0  Trainee  questionnaire.  Twenty-one  (21)  MOS  63W10 
students  who  had  received  training  with  the  Seville/ 

Burtek  simulator  completed  this  questionnaire  after 
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they  had  been  tested  on  their  ability  to  perform  a 
troubleshooting  task  on  operational  equipment.  The 
questionnaire  was  also  completed  by  46  students  who 
had  been  trained  with  the  simulator,  but  had  not 
been  tested  on  their  ability  to  perform  a  trouble¬ 
shooting  task  on  the  operational  equipment  due  to 
constraints  encountered  during  the  course  of  the 
experiment. 

0  Structured  i ntervi ew.  An  interview  was  conducted 
with  a  63WlO  MOS  instructor.  In  addition,  three 
of  the  individuals  interviewed  in  Experiment  1 
(two  course  developers  and  the  on-site  data  collector) 
made  comments  that  were  applicable  to  the  63W10  MOS. 

Relevant  comments  made  by  these  individuals  are 
reported  in  this  experiment. 

Materials.  The  questionnaires  and  structured  interview  form  used  in 
this  experiment  were  identical  to  those  used  in  Experiment  1  (see  Appendix  A). 

Procedure .  The  procedure  used  in  Experiment  2  was  highly  similar  to 
that  used  in  the  first  experiment.  It  can  be  summarized  as  follows: 

0  Subjects  were  briefed  before  responding  to 
questionnaires  or  interview  questions. 

0  Initial  instructor  questionnaires  were  com¬ 
pleted  in  a  group  setting  after  instructors 
were  familiar  with  the  simulator. 

0  An  instructor  questionnaire  was  completed 
for  each  of  22  tasks  addressed  by  simulator 
lessons. 

0  Course  developer  questionnaires  were  not 
completed. 

0  Trainees  responded  to  questionnaires  after 
training  sessions  were  completed  (for  trainees 
who  were  not  subsequently  tested  on  operational 
equipment)  or  after  training  and  testing  v;ere 
completed. 

0  Structured  interviews  were  conducted  on  an 
individual  basis  after  transfer-of-training 
data  had  been  collected. 

Respondents  commented  on  specific  features  of  the  Seville/Burtek  device- 
These  features  are  described  in  Table  2.  The  3-D  module  on  which  the  MOS 
63W10  students  were  trained  was  identical  to  the  3-D  module  on  which  the 
63B30  students  were  trained. 
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Results 


Initial  Instructor  Questionnaire.  As  indicated  by  the  responses  to 


Question  1,  63WT0  instructors  experienced  few  problems  learning  to  operate 
the  simulator. 

QUESTION  1:  How  easy  or  hard  is  it  to  learn  to  use  the  simulator? 
I^TINGS:  1  =  very  hard,  5  =  very  easy 
N  =  3 


Response 


VERY  EASY 
FAIRLY  EASY 
AVERAGE 
FAIRLY  HARD 
VERY  HARD 


Number  of  Responses 


TOTAL 


The  mean  rating  for  all  instructors  was  4.33  (standard  deviation  =  .58). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

One  instructor  stated  that  the  training  sessions  conducted  by  representa¬ 
tives  of  Seville/Burtek  were  very  helpful.  The  other  instructors  felt  that 
the  procedures  for  operating  the  simulator  could  be  learned  easily  since 
these  procedures  were  not  complicated. 

QUESTION  3:  How  do  you  feel  about  the  simulator? 

RATINGS:  1  =  dislike  it  very  much,  5  =  like  it  very  much 

N  =  3 


Response 

LIKE  IT  VERY  MUCH 
LIKE  IT  SOMEWHAT 
NO  STRONG  FEELINGS  ABOUT  IT 
DISLIKE  IT  SOMEWHAT 
DISLIKE  IT  VERY  MUCH 


Number  of  Responses 


TOTAL 


The  mean  rating  for  all  instructors  was  4.33  (standard  deviation  -  .58). 
QUESTION  4:  Why  did  you  answer  Question  3  as  you  did? 

All  instructors  liked  the  simulator  because  it  was  easy  to  operate. 


QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

One  instructor  was  initially  skeptical  about  the  device,  but  after  com¬ 
pleting  the  operator's  training  course,  he  had  a  favorable  opinion  of  the 
simulator  because  it  was  simple  to  operate. 


Instructor  Questionnaire.  One  instructor  and  one  student  worked 
together  to  complete  22  lessons  presented  on  the  Seville/Burtek  simulator. 
The  instructor  filled  out  a  questionnaire  after  each  lesson  had  been  com¬ 
pleted.  Responses  to  the  five  questions  that  appeared  on  the  22  question¬ 
naires  are  sumnarized  below. 

QUESTION  1:  How  well  does  the  simulator  train  this  task? 

RATINGS:  1  «  very  poorly,  5  *  very  well 

N  *=  1 


Response 


Number  of  Responses 


VERY  WELL  19 
MODERATELY  WELL  1 
FAIR  2 
POORLY  0 
VERY  POORLY  0 


TOTAL  71 


The  mean  rating  for  all  tasks  was  4.77  (standard  deviation  *  .61).  Raw 
data  are  presented  in  Table  C-1. 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Other  than  indicating  that  the  exercises  were  "very  good,"  the  instructor 
made  no  specific  comments  about  why  tasks  were  rated  so  highly.  He  indicated 
that  lower  ratings  were  given  to  tasks  that  were  redundant  or  too  simple. 

QUESTION  3:  What  specific  features  of  the  simulator  (involved  in  this 
task)  do  you  like  or  dislike? 

The  instructor  found  it  easy  to  follow  the  lessons  and  liked  the  clean 
work  environment  (i.e.,  no  oil,  dirt,  etc.). 


QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

The  instructor  made  no  relevant  responses  to  these  questions. 

Trainee  Questionnaire.  Responses  of  the  21  students  who  were  tested  on 
operational  equipment  are reported  separately  from  the  responses  of  the  46 
students  who  were  not  tested  on  operational  equipment. 

QUESTION  1:  How  do  you  feel  about  the  simulator? 

RATINGS:  1  *  dislike  it  very  much,  5  “  like  it  very  much 


Response 


Number  of  Responses 


Students  Tested  Students  Not  Tested 


(N  =  21)  (N  «  46) 


LIKE  IT  VERY  MUCH  10  31 
LIKE  IT  SOMEWHAT  8  10 
NO  STRONG  FEELINGS 

ABOUT  IT  1  3 
DISLIKE  IT  SOMEWHAT  2  0 
DISLIKE  IT  VERY  MUCH  0  2 

TOTAL  TOTAL  46 


The  mean  rating  for  students  who  were  tested  was  4.24  (standard  devia¬ 
tion  .94).  The  mean  rating  for  students  who  were  not  tested  was  4.48 
(standard  deviation  »  .96). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Students  who  were  tested  on  operational  equipment  and  rated  the  simula¬ 
tor  highly  did  so  for  a  number  of  reasons.  Step-by-step  instructions  for 
completing  lessons  and  performance  feedback  were  features  most  frequently 
noted.  Students  also  liked  the  simulator  because  it  was  easy  to  use,  safe, 
fun  to  operate,  and  clean.  One  student  who  did  not  rate  the  simulator  highly 
felt  that  it  was  not  appropriate  for  use  with  entry  level  soldiers.  Other 
students  who  indicated  that  they  disliked  the  simulator  provided  no  specific 
explanation  for  their  opinions. 

The  41  students  who  were  not  tested  on  operational  equipment,  and  who 
rated  the  simulator  highly,  liked  the  device  because  it  was  faster,  cleaner, 
safer,  and  easier  tn  operate  than  actual  equipment.  They  also  liked  the 
device  because  it  provided  feedback  and  useful  illustrations  on  the  slide 
projector  unit.  The  two  students  who  gave  the  simulator  a  poor  rating 
stated  that  they  did  not  understand  how  to  operate  the  device. 

QUESTION  3:  What  specific  features  of  the  simulator  do  you  like  or 
dislike? 

Students  who  were  tested  on  their  ability  to  troubleshoot  a  task  on 
operational  equipment  liked  the  hardcopy  printout,  feedback,  and  the  slide 
projector  unit.  These  students  did  not  like  the  sensitivity  of  the  3-D 
module,  the  low  fidelity  representation  of  certain  (unspecified)  components, 
and  the  confusion  that  results  from  trying  to  attend  to  mar\y  sources  of 
information. 

Students  who  were  not  tested  on  operational  equipment  responded  to  this 
question  in  a  manner  similar  to  students  who  were  tested.  They  liked  the 
hardcopy  printout,  feedback,  and  the  slide  projector  unit.  These  students 
also  liked  the  fact  that  the  3-D  module  looked,  sounded,  and  functioned  like 
a  diesel  engine.  Two  students  liked  the  fact  that  the  simulator  was  simple 
to  operate.  Students  disliked  the  sensitivity  of  the  3-D  module. 

QUESTION  4:  Why  do  you  like  or  dislike  these  features? 


25 


Students  In  both  groups  liked  the  hardcopy  printout  and  feedback  because 
these  features  allowed  them  to  determine  when  they  were  committing  errors. 
Students  liked  the  slide  projector  unit  because  the  diagrams  and  photographs 
presented  on  this  unit  were  superior  to  the  materials  presented  in  the  TMs. 

The  sensitivity  of  the  3-0  module  was  disliked  by  students  because  its  sensors 
recorded  minor  movements  and  vibrations  as  student  errors. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

Many  students  (both  tested  and  untested)  enjoyed  working  with  the  simu¬ 
lator.  They  also  stated  that  they  desired  additional  training  with  the 
simulator. 

Structured  Interview.  Responses  to  the  13  questions  that  were  posed 
during  the  structured  interview  are  presented  below. 

QUESTION  1:  What  were  the  instructional  features  of  the  simulator  that 
were  applicable  to  the  school's  training  course? 

The  63W10  MOS  instructor  stated  that  the  3-D  module  was  applicable  to 
the  school's  training  course  since  it  allowed  students  to  gain  hands-on 
experience  during  training.  He  stated  that  the  ability  to  quickly  insert 
malfunctions  into  the  3-0  module  was  one  of  the  outstanding  features  of  this 
simulator. 

He  also  felt  that  feedback  provided  to  students  and  instructors  was  very 
useful  since  these  features  allowed  students  to  complete  .lessons  with  minimal 
instructor  intervention.  This  comment  contrasts  with  the  statement  of  the 
interviewee  in  Experiment  1  (page  13)  who  felt  that  feedback  confused 
students  at  times. 

Other  features  of  the  simulator  that  the  instructor  found  to  be  applicable 
include  flexible  arrangement  of  lessons,  remove/ replace  capability,  editing 
system,  slide  projector  unit,  self-paced  method  of  instruction,  and  random 
malfunction  selection.  Most  of  the  interviewees  in  Experiment  1  did  not 
think  the  random  malfunction  selection  feature  was  applicable. 


Ratings  made  by  the  instructor  are  presented  below: 


RATINGS 


[1  °  None,  7  =  Highest) 


Rate  the  training  value 
of  the  features  as  a 
whole. 


Rate  the  potential 
training  value  of  the 
features  as  a  whole. 


N  RATING 

1  6 

1  7 


QUESTION  2:  What  were  the  instructional  features  of  the  simulator  that 
were  not  applicable  to  the  school's  training  course? 

The  instructor  stated  that  the  lessons  involving  the  STE/ICE  kit  were 
inapplicable  because  they  did  not  include  instructions  for  set  up  and  check  out 


of  this  equipment.  He  noted  that  it  was  particularly  important  for  MOS  63W10 
students  to  learn  the  set>up/check>out  procedure  because  they  had  no  previous 
experience  with  STE/ICE. 

Two  individuals  interviewed  during  Experiment  1  made  responses  to  this 
question  that  are  appropriate  for  inclusion  in  this  experiment.  The  on-site 
data  collector  felt  that  the  3-D  module  was  not  applicable  to  the  63W10  MOS 
because  the  module  simulated  an  engine  that  was  not  addressed  in  the  curricu¬ 
lum  of  conventionally  trained  students.  A  course  developer  stated  that  the 
remove/replace  capability  was  of  limited  value  to  63W10  students  since  they 
are  primarily  involved  with  troubleshooting  tasks. 

QUESTION  3:  Which  features  of  the  lesson  presentation  helped  make  the 
simulator  lessons  interesting  to  the  students? 

The  instructor  indicated  that  there  were  several  features  that  maintained 
students'  interest  in  the  lesson.  He  stated  that  students  found  the  3-D 
module  (and  associated  sound  effects)  very  interesting.  The  instructor 
felt  that  high  physical  and  functional  fidelity  and  low  probability  of  student 
injury  were  factors  that  contributed  to  the  students'  interest  in  the  3-D 
module.  Information  presented  on  the  student  CRT  was  seen  as  maintaining 
student  interest  since  this  information  directed  students  to  perform  specific 
actions,  and  it  provided  them  with  feedback  about  these  actions.  The 
instructor  felt  that  students  were  interested  in  the  materials  presented  by 
the  slide  projector.  He  added,  however,  that  students  lost  interest  in 
lessons  when  this  unit  malfunctioned.  Self-paced  lesson  presentation  was  an 
additional  feature  mentioned  by  the  instructor.  He  noted  that  in  conventional 
training,  students  lose  interest  in  lessons  when  instructors  present  material 
to  a  group  of  students  at  a  rate  suited  only  to  the  most  advanced  student. 

The  instructor  felt  that  this  problem  would  not  occur  with  the  self-paced 
lessons  presented  on  the  Seville/Burtek  device. 

Ratings  made  by  the  instructor  are  presented  below: 

RATINGS 

(1  «  None,  7  «  Greatest)  ^  RATING 

Rate  the  training  value 

of  these  features  as  a 

whole.  1  7 

Rate  the  potential 

training  value  of  these 

features  as  a  whole.  1  7 

QUESTION  4:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  particularly  effective? 

Although  the  instructor  had  completed  most  of  the  lessons  presented  by 
the  training  device,  he  stated  that  he  did  not  recall  the  details  of  the 
lessons  well  enough  to  identify  those  that  were  particularly  effective. 


Ratings  made  by  the  Instructor  are  presented  below: 


RATINGS 


[1  «  None.  7  «  Greatest) 


Rate  the  training  value 
of  these  lessons  as  a  whole. 


N 

1 


Rate  the  potential 
training  value  of  these 
lessons  as  a  whole.  1 


RATING 

7 

7 


QUESTION  5:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  Ineffective? 


The  Instructor  stated  that  lessons  Involving  the  STE/ICE  kit  need 
Improvement.  He  also  Indicated  that  certain  (unspecified)  lessons  would  be 
more  effective  If  they  were  more  detailed. 

QUESTION  6:  Were  there  any  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  being  presented  by  the  simulator  was  above  the  students? 

The  MOS  63W10  Instructor  felt  that  students  experienced  difficulty  using 
TMs.  He  Indicated  that  many  of  the  problems  students  experienced  In  this 
area  were  due  to  the  students'  poor  reading  skills.  This  same  conrient  about 
MOS  63W10  students  was  made  by  one  of  the  Individuals  Interviewed  In  Experi¬ 
ment  1.  Another  Individual  Interviewed  In  Experiment  1  stated  that  the 
material  presented  by  the  simulator  was  more  appropriate  for  students  with 
more  advanced  skills  (MOS  63B30}  than  It  was  for  students  with  basic  skills 
(MOS  63W10). 


The  63W10  Instructor  also  stated  that  the  students  found  It  difficult  to 
operate  the  simulator.  He  Indicated  that  problems  experienced  by  students 
operating  the  device  arose  because  students  were  provided  too  brief  an 
orientation  to  the  simulator.  This  orientation  was  abbreviated  because  School 
administrators  required  students  to  complete  training  In  the  least  amount  of 
time  feasible. 


QUESTION  7:  Were  there  any  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  being  presented  by  the  simulator  should  be  Increased? 

The  Instructor  felt  that  students  would  become  confused  If  any  of  the 
lessons  were  more  difficult. 


QUESTION  8:  What  hardware  features  of  the  simulator  made  It  more 
effective  than  conventional  training? 

Four  hardware  features  were  mentioned  by  the  instructor.  The  3-D 
•jule  was  thought  to  be  more  effective  than  operational  equipment  because 
students  have  easy  access  to  components,  and  because  students  can  learn  to 
troubleshoot  the  electrical  system  without  fear  of  receiving  an  electrical 
shock.  The  Instructor  did  not  Indicate  that  the  benefits  of  easy  access  to 
components  may  have  been  offset  by  the  low  physical  fidelity  of  the  3-D 
module  (as  was  Indicated  by  the  Interviewees  In  Experiment  1). 


The  process  of  entering  decisions  into  the  student  responder  panel  was 
described  as  more  effective  than  conventional  training  methods  because 
simulator-trained  students  are  forced  to  enter  their  decisions  into  this 
unit,  whereas  students  trained  conventionally  may  not  be  cognizant  of  the 
specific  decisions  they  make  when  troubleshooting.  This  comment  does  not 
indicate  that  the  student  responder  panel  is  easy  to  use.  Rather  it 
indicates  that  this  unit  requires  students  to  think  about  the  troubleihooting 
process  and  make  troubleshooting  decisions  in  order  to  progress  through  a 
lesson.  Students  may  progress  through  conventional  training  without  giving 
much  thought  to  the  troubleshooting  process.  The  instructor  felt  that  the 
information  provided  by  the  instructor  CRT  and  the  hardcopy  printout  made 
the  simulator  more  effective  than  conventional  training  since  they  allowed 
the  instructor  to  monitor  actions  performed  by  the  student. 

Ratings  made  by  the  instructor  are  presented  below: 


RATINGS 

(1  ■  None,  7  ■  Greatest) 

N 

RATING 

Rate  the  training  value  of 
these  features  as  a  whole. 

1 

4 

Rate  the  potential  training 
value  of  these  features  as 
a  whole. 

1 

5 

QUESTION  9:  What  were  the  features  that  made  this  simulator  easy  for 
the  instructor  to  operate? 

The  instructor  stated  that  the  simulator  could  be  operated  easily  by 
entering  commands  on  the  instructor  panel.  He  felt  that  malfunctions  could 
be  inserted  easily  into  the  3-0  module  and  that  the  device  allowed  great 
flexibility  in  the  sequence  in  which  lessons  were  presented.  He  noted  that 
the  limited  durability  of  remove/ replace  components,  and  the  poor  reliability 
of  the  slide  projector  unit  made  the  simulator  difficult  to  operate. 

Ratings  made  by  the  instructor  are  presented  below: 


RATINGS 

(1  =  None,  7  «  Greatest) 

N 

RATING 

Rate  the  training  value  of 
these  features  as  a  whole. 

1 

6 

Rate  the  potential  training 
value  of  these  features  as 
a  whole. 

1 

7 

QUESTION  10:  What  were  the  features  that  made  this  simulator  easy  for 
the  students  to  operate? 

The  instructor  felt  that  the  student  guidebook  and  the  accessibility  of 
components  on  the  3-D  module  made  the  simulator  easy  for  students  to  operate. 


Ratings  made  by  the  instructor  are  presented  below: 


RATINGS 


[1  *  None,  7  ■  Greatest! 


Rate  the  training  value  of 
these  features  as  a  whole. 


Rate  the  potential  training 
value  of  these  features  as 
a  whole. 


N 

1 

1 


RATING 

5 

A 


QUESTION  11:  What  aspects  of  the  device  would  be  appropriate  for  sub¬ 
stituting  for  actual  equipment? 

The  instructor  felt  that  the  3-D  module  could  be  substituted  for  actual 
equipment.  The  ease  of  inserting  malfunctions  into  this  module  was  mentioned 
as  being  one  of  its  most  valuable  features.  The  instructor  stated  that  sub¬ 
stitution  of  the  3-D  module  for  operational  equipment  should  not  be  complete. 
He  felt  that  students  also  needed  exposure  to  operational  equipment  during 
training. 

Other  features  that  the  instructor  felt  were  substitutable  for  opera¬ 
tional  equipment  include  the  hardcopy  printout,  slide  projector  unit,  and 
flexible  lesson  presentation. 

Ratings  made  by  the  instructor  are  presented  below: 


RATINGS 


[1  *  None,  7  «  Greatest] 


Rate  the  training  value  of 
these  features  as  a  whole. 


Rate  the  potential  training 
value  of  these  features  as 
a  whole. 


N 

1 

1 


RATING 

6 

7 


QUESTION  12:  What  types  of  problems  did  students  have? 

The  instructor  felt  that  the  most  serious  problem  experienced  by  students 
was  the  fragile  nature  of  components  that  were  removed/ replaced  from  the  3-D 
module.  He  indicated  that  students  were  not  trained  to  apply  appropriate 
amounts  of  force  when  removing/replacing  components.  Further,  he  noted  that 
time  required  to  repair  damaged  helical  coils,  bolts,  electrical  contacts, 
and  fittings  decreased  the  amount  of  time  available  to  train  students. 

QUESTION  13:  How  would  you  employ  the  simulator  in  order  to  gain  maxi¬ 
mum  benefit  from  it? 


The  instructor  stated  that  the  Seville/Burtek  simulator  would  be  maxi¬ 
mally  effective  if  it  were  used  in  conjunction  with  operational  equipment  to 
train  10,  20,  and  30  level  students.  He  felt  that  the  student/instructor 
ratio  should  be  no  greater  than  two  to  one.  The  instructor  indicated  that 
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the  device  would  be  appropriate  for  the  direct  support  and  organizational 
levels  of  maintenance,  but  not  for  the  general  support  and  depot  levels  of 
maintenance. 

Ratings  made  by  the  instructor  for  the  device  as  a  whole  are  presented 
below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

RATING 

Rate  the  training  value 
of  the  simulator  as  a  whole. 

1 

6 

Rate  the  potential  training 
value  of  the  simulator  as  a 

whole. 

1 

6 

Data  Collector  Observations.  Information  about  the  operation  of  the 
simulator  recorded  by  the  data  collector  is  identical  for  63W10  and  63B30 
students.  Since  this  data  has  been  reported  for  63B30  students  in  Experi¬ 
ment  1,  it  will  not  be  reported  for  63W10  students. 

Discussion 


A  series  of  positive  and  negative  statements  about  the  Seville/Burtek 
device  are  presented  in  light  of  the  data  presented  above. 

Positive  Statements.  Instructors  and  trainees  hold  favorable  opinions 
of  the  Seville/Burtek  device^  Instructors  -indicated  that  the  device  was  easy 
to  operate  and  that  it  did  a  very  good  job  of  training  students  to  perform 
troubleshooting  tasks.  Students  stated  that  they  enjoyed  working  with  the 
simulator  and  that  they  desired  additional  training  with  the  simulator. 

Students  and  the  instructor  liked  a  number  of  device  features.  Students 
repeatedly  stated  that  they  enjoyed  working  on  the  simulator  because  it  pro¬ 
vided  performance  feedback,  step-by-step  instructions  for  accomplishing  tasks, 
and  a  slide  projector  ur.it  that  helped  them  to  identify  engine  components. 

The  63W10  instructor  liked  many  features,  in  particular,  the  device's  per¬ 
formance  feedback  and  the  ease  of  inserting  malfunctions  into  the  3-D  module. 

Negative  Statements.  Lessons  should  have  included  procedures  for  set  up 
and  check  out  of  the  STE/ICE  kit.  The  63W10  instructor  felt  that  the  lack  of 
set-up  and  check-out  procedures  for  STE/ICE  impaired  student  performance 
because  students  had  no  prior  experience  with  STE/ICE. 

The  3-D  module  is  too  sensitive.  Students  and  the  instructor  disliked 
the  fact  that  minor  vibrations  that  normally  accompany  remove/ replace  tasks 
were  recorded  as  errors. 

The  durability  of  the  simulator  is  low.  The  63W10  instructor  felt  that 
the  device  could  not  withstand  rigorous  use. 


Experiment  3 

Twenty-two  (22)  students  from  the  63H30  MOS  (Direct  Support  Maintenance 
Supervisor)  were  trained  to  troubleshoot  the  starting  and  charging  system  of 
an  M110A2  self-propelled  howitzer  using  either  conventional  methods  (12 
students)  or  the  Grumman  simulator  (10  students).  Students  were  then  tested 
on  their  ability  to  perform  this  task  on  operational  equipment.  Data  consist 
of  responses  to  questionnaires  and  structured  interviews,  as  well  as  observa¬ 
tions  made  by  the  on-site  data  collector  during  the  course  of  the  experiment. 

Method 

Subjects.  Characteristics  of  subjects  who  provided  data  are  as  follows: 

0  Initial  instructor  questionnaire.  Five  NCOS  who  were 
instructors  for  the  63H30  and  63D30  MOSs  completed  the 
questionnaire. 

0  Instructor  questionnaire.  Questionnaires  were  completed 
for  each  of  the  lesson  segments  presented  by  the  device. 

See  Table  4  for  a  description  of  lesson  segments.  Four 
NCOS  who  were  instructors  for  the  63H30  and  63D30  MOSs 
completed  questionnaires.  In  order  to  collect  additional 
data,  two  other  individuals  knowledgeable  about  the  device 
(a  civilian  training  specialist  and  the  project  liaison 
officer)  also  completed  the  questionnaire. 

0  Course  developer  questionnaire.  This  form  was  completed 
by  an  individual  responsible  for  creating  and  modifying 
curricula  for  the  63H30  and  63D30  MOSs.  In  order  to 
collect  additional  data,  this  form  was  also  completed 
by  two  other  individuals  knowledgeable  about  the  device 
(a  civilian  training  specialist  and  the  project  liaison 
officer) . 
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Trainee  questionnaire.  Ten  63H30  students  who  received 
training  with  the  Grumman  device  completed  this  question¬ 
naire  after  they  were  tested  on  their  ability  to  perform 
a  troubleshooting  task  on  operational  equipment.  The 
questionnaire  was  also  completed  by  five  students  who 


were  trained  with  the  simulator,  but  were  not  tested 


on  their  ability  to  perform  a  troubleshooting  task  on 
operational  equipment. 


0  Structured  interview.  Three  instructors  for  the  63H30 
and  63D30  MOSs,  the  project  liaison  officer,  and  the 
on-site  data  collector  were  interview?'"! . 


Materials.  The  questionnaires  and  structured  interview  form  used  in  this 
experiment  were  identical  to  those  used  in  Experiments  1  and  2.  (See 
Appendix  A.) 
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Grumman  Simulator  Lessons  and  Segments 


Segments 


0:  Introduction,  Part  1 

1:  Introduction,  Part  2 

2:  Vehicle  Test  Meter  (VTM)  Set-up  and  Check-out 
Tutorial 

3:  VTM  Set-up  and  Check-out  Exercise 

4:  Introduction  to  the  Starting  System 

5:  Starting  System  Problem,  Part  1 
(VTM  Set  up  and  Check  out) 

6:  Starting  System  Problem,  Part  2 

(Troubleshooting  a  Defective  Transmission 
Neutral  Position  Switch) 

7:  Charging  System  Problem  1 
(Defective  Lead  1) 

8:  Charging  System  Problem  2 
(Defective  Voltage  Regulator) 

9:  Charging  System  Problem  3 
(Defective  Generator) 
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Procedure.  The  procedure  followed  for  Experiment  3  was  similar  to  the 
procedure  reported  for  the  first  two  experiments.  The  procedure  is  summarized 
below: 

0  Subjects  were  briefed  before  responding  to  ques¬ 
tionnaires  or  interview  questions. 

0  Initial  instructor  questionnaires  were  completed 
in  a  group  setting  after  instructors  were  familiar 
with  the  simulator. 

0  Subjects  were  asked  to  complete  an  instructor 
questionnaire  for  each  of  the  nine  segments 
listed  in  Table  4. 

0  Subjects  were  asked  to  complete  course  developer 
questionnaires  for  each  of  the  nine  segments 
listed  in  Table  4. 

0  Trainees  responded  to  questionnaires  after  training 
sessions  were  completed  (for  trainees  who  were  not 
subsequently  tested  on  operational  equipment)  or 
after  training  and  testing  were  completed. 

0  Structured  interviews  were  conducted  on  an  individual 
basis  after  transfer-of-training  data  had  been 
collected. 

Several  questions  asked  respondents  to  comment  on  specific  features  of 
the  Grumman  simulator.  In  order  to  clarify  responses  to  these  questions, 
brief  descriptions  of  the  major  features  of  the  Grumman  device  are  presented 
in  Table  5. 

Results 


Initial  Instructor  Questionnaire.  Responses  to  this  questionnaire  are 
summarized  below. 

QUESTION  1:  How  easy  or  hard  is  it  to  learn  to  use  the  simulator? 
RATINGS:  1  =  very  hard,  5  =  very  easy 
N  =  5 


Response 


Number  of  Responses 


VERY  EASY  2 
FAIRLY  EASY  0 
AVERAGE  3 
FAIRLY  HARD  0 
VERY  HARD  0 


TOTAL  5 


The  mean  rating  for  all  instructors  was  3.8  (standard  deviation  =  1.1). 


Table  5 


Description  of  Grunman  Device  Features 


Feature 

Description 

video  disc  system 

still  frames,  mrtlon  frames,  and  comouter-generated 
text  were  presented  by  this  systam  on  the  student 

CRT  unit  In  order  to  direct  student  actions  and 
provide  feedback. 

Touch  panel 

Students  entered  their  decisions  Into  the  training 
device  by  touching  certain  locations  on  the  student 

CRT  that  displey  words  (yes,  no,  etc.),  pictures 
(master  twitch.  Instrument  twitch,  etc.)  or  schematics. 

Request  help 

Hany  frames  presented  by  the  video  disc  player 
ellow  students  to  request  help  by  touching  the  CRT. 

Help  Is  presented  In  the  form  of  audio  end/or  visual 
cues. 

Repeat  lesson 

At  certain  predetermined  points  In  the  training  pro¬ 
gram,  students  may  choose  to  repeat  segments,  or 
parts  of  segments. 

Call  Instructor 

When  students  make  two  consecutive  errors,  the  kjvlce 
ceases  to  accept  student  input  and  the  student  receives 
a  message  to  call  an  instructor. 

Sound  effects 

Various  engine  sounds  (cranking.  Idle,  shut-down)  were 
simulated  In  the  H1110A2  howitzer  configuration. 

Hardcopy  printout 

A  record  of  student  performance  could  be  printed  after 
e  lesson  was  completed. 

Automated  pre-lesson 
check 

Prior  to  starting  a  lesson,  the  device  checks  to 
ensure  that  ell  switches,  cables,  etc.  of  the  3-0 
module  are  In  the  correct  configuration.  Instruc¬ 
tions  for  correcting  erroneous  configurations  are 
presented  on  the  student  CRT. 

Lesson  arrangement 

Students  normally  conplete  lessons  In  a  fixed  sequence. 

The  training  device  keeps  track  of  the  segments  that  a 
student  has  completed,  presenting  the  appropriate  seg¬ 
ment  each  time  a  student  works  with  the  device. 

Universal  Instructor 

This  feature  allows  Instructors  to  present  segments 

In  any  sequence;  however,  no  record  of  student  per¬ 
formance  Is  kept  when  this  feature  is  enabled. 

Instructor  CRT 

Information  about  the  video  disc  system  Is  presented 
on  the  Instructor  CRT  iHien  students  parti  clpete  In 
lessons  on  the  simulator. 

Nott:  The  }-D  aodult  of  the  Crueiwn  slnulitor  for  the  63H3D  tnd  63D30  NOSs 
(Experiments  3  ind  4)  consists  of  controls,  dlspli/s,  test  equlpnent, 
ind  components  of  the  starting  end  charging  system  for  the  H110A2 
self-propelled  howitzer. 

The  3-0  module  of  the  Grunnan  device  used  by  NOSs  E4E,  Z4G,  and  24R 
(Experiment  6)  Is  a  full-scale  reproduction  of  an  Ixproved  Hawk  High 
Power  niixninator  Radar  transmitter  unit.  The  3-0  module  Is  composed 
of  a  cabinet,  ^■'terlor  components  (cables,  power  supplies,  etc.), 
controls,  displays,  and  test  equipment. 
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QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Instructors  indicated  that  although  procedures  for  operating  the  device 
were  not  complicated,  device  malfunctions  detracted  from  instructor  training 
sessions. 

QUESTION  3:  How  do  you  feel  about  the  simulator? 

WTINGS:  1  *  dislike  it  very  much,  5  “  like  it  very  much 

N  =  5 

Response  Number  of  Responses 

LIKE  IT  VERY  MUCH  0 

LIKE  IT  SOMEWHAT  2 

NO  STRONG  FEELINGS  ABOUT  IT  1 

DISLIKE  IT  SOMEWHAT  1 

DISLIKE  IT  VERY  MUCH  1 

TOTAL  F 

The  mean  rating  for  all  instructors  was  2.8  (standard  deviation  »  1.3). 

QUESTION  4;  Why  did  you  answer  Question  3  as  you  did? 

The  instructors  generally  agreed  that  the  lessons  presented  on  the  sim¬ 
ulator  were  too  simple  for  the  63H30  students.  They  suggested  that  it  may  be 
more  appropriate  to  use  the  simulator  to  train  entry  level  students.  One 
instructor  disliked  the  simulator  very  much  because  the  lessons  did  not 
adequately  describe  how  to  use  the  STE/ICE  kit  during  troubleshooting  tasks. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

Most  of  the  instructors  indicated  that  the  lessons  did  not  adequately 
describe  how  to  use  the  STE/ICE  kit  when  troubleshooting.  They  indicated 
that  the  lessons  showed  students  how  the  STE/ICE  kit  could  be  used  as  a 
volt/ohm  meter,  but  they  did  not  describe  the  other  capabilities  of  this 
equipment. 

Instructor  Questionnaire.  Data  reported  for  the  instructor  questionnaire 
are  based  on  the  responses  of  instructors  and  other  individuals  knowledgeable 
about  the  Grumman  simulator  and  the  63H30  MOS.  The  tasks  rated  by  instructors 
are  refered  to  by  simulator  lesson  segments. 

QUESTION  1:  How  well  does  the  simulator  train  this  task? 

RATINGS;  1  =  very  poorly,  5  =  very  well 

N  =  6 


Response 

Number  of  Responses 

VERY  WELL 

7 

MODERATELY  WELL 

12 

FAIR 

9 

POORLY 

13 

VERY  POORLY 

4 

TOTAL  45 
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The  mean  rating  for  all  lesson  segments  was  3.1  (standard  deviation  = 

1.25).  Raw  data  are  presented  in  Table  D-1. 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Respondents  provided  low  ratings  for  those  segments  (6,  7,  8,  and  9)  in 
which  the  simulator  repeatedly  "locked  up"  (would  not  register  student  actions 
or  advance  to  the  next  step).  Respondents  gave  low  ratings  to  segments  that 
presented  incorrect  procedures  and  segments  that  progressed  at  a  slow  rate. 

The  first  few  segments  (0-3)  were  rated  relatively  highly  because  they  were 
well -prepared. 

QUESTION  3:  What  specific  features  of  the  simulator  (involved  in  this 
task)  do  you  like  or  dislike? 

Respondents  liked  the  instructor  station,  the  3-D  module,  the  color 
video  displays,  the  audio  track  that  accompanied  the  video  presentation,  the 
fact  that  students  could  make  errors  without  damaging  equipment,  and  the 
interaction  between  the  student  and  simulator.  Respondents  disliked  the  fre¬ 
quency  with  which  the  device  malfunctioned,  the  low  fidelity  of  the  3-D 
module,  the  rate  at  which  lessons  were  presented,  the  inability  to  skip  parts 
of  a  lesson,  the  inability  to  repeat  lessons,  and  various  procedural  errors 
in  the  lessons. 

QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

One  respondent  liked  the  interaction  between  the  student  and  the  simula¬ 
tor  because  it  promoted  learning.  Respondents  who  liked  the  3-D  module 
indicated  that  it  was  realistic  and  gave  students  "hands-on"  troubleshooting 
experience.  Other  respondents,  however,  disliked  aspects  of  the  3-D  module. 

One  respondent  felt  that  the  W5  cable  of  the  STE/ICE  kit  was  not  a  high 
fidelity  representation  of  the  operational  equipment.  Another  respondent 
stated  that  it  was  difficult  to  understand  the  interrelationship  of  components 
found  on  the  simulator.  Other  (unspecified)  features  were  disliked  because 
they  wasted  time  and  did  not  promote  learning. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

One  respondent  noted  that  in  Segment  2,  students  were  not  instructed  to 
visually  inspect  the  VTM  (a  STE/ICE  component)  prior  to  hook-up.  This 
respondent  also  stated  that  the  simulator  needed  to  be  completely  revised. 
Another  respondent  felt  that  students  accepted  the  lessons  because  they  could 
interact  with  the  simulator.  One  respondent  stated  that  the  simulator  was  a 
very  poorly  designed  piece  of  equipment.  One  respondent  felt  that,  for 
Segment  7,  the  task  was  taught  incorrectly  and  would  confuse  students.  Another 
respondent  stated  that  students  were  not  taught  how  to  start  the  engine  in 
Segment  1. 

Course  Developer  Questionnaire.  Responses  were  made  by  one  course 
developer  and  two  other  individuals  who  were  knowledgeable  about  the  Grumman 
device. 
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QUESTION  1:  Is  this  a  critical  task  which  needs  to  be  trained? 
RESPONSES:  Yes,  No 
N  =  3 


Respondents  indicated  that  all  segments,  except  Segments  0  and  6, 
addressed  critical  tasks  that  should  be  trained.  Segment  0  is  an  introduction 
to  AMTESS,  while  Segment  6  addresses  troubleshooting  a  defective  transmission 
neutral  position  switch.  Raw  data  are  presented  in  Table  D-2. 

QUESTION  2:  Is  this  task  currently  trained  on  operational  equipment? 

RESPONSES:  Yes,  No 

N  =  3 


Respondents  agreed  that  Segment  0  (Introduction  to  AMTESS)  and  Segment  9 
(Charging  System  Problem  3)  were  not  taught  on  operational  equipment. 
Respondents  did  not  agree  on  whether  Segments  6  and  7  were  taught  on  opera¬ 
tional  equipment.  Raw  data  are  presented  in  Table  D-3. 

QUESTION  3:  How  difficult  is  it  to  perform  this  task? 

RATINGS:  1  «  very  easy,  5  =  very  difficult 
N  =  3 


Response 


Number  of  Responses 


VERY  DIFFICULT  0 
SOMEWHAT  DIFFICULT  2 
ABOUT  AVERAGE  IN  DIFFICULTY  6 
SOMEWHAT  EASY  6 
VERY  EASY  5 


TOTAL  T? 


The  mean  rating  for  all  tasks  was  2.26  (standard  deviation  =  .99).  Raw 
data  are  presented  in  Table  D-4. 


QUESTION  4:  At  what  skill  level  should  a  trainee  perform  this  task  at 
the  end  of  training? 

RATINGS:  1  =  novice,  5  *  expert 
N  =  3 
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Response 


Number  of  Responses 


EXPERT  7 
LESS  THAN  EXPERT  4 
APPRENTICE  4 
BETTER  THAN  NOVICE  3 
NOVICE  1 


TOTAL  19 


The  mean  rating  for  all  tasks  was  3.68  (standard  deviation  =  1.29).  Raw 
data  are  presented  in  Table  D-5. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

One  respondent  made  five  comments  about  Segment  5: 

0  Form  DA  2404  (Equipment  Inspection  and  Maintenance 
Worksheet)  appears  on  the  student  CRT  and  indicates 
that  the  engine  will  not  crank.  However,  the  3-D 
module  can  be  cranked  by  students. 

0  Electrical  wiring  on  the  3-D  module,  including  STE/ICE, 
should  be  labeled  clearly. 

0  The  lessons  should  be  updated  to  reflect  a  new  TM 
(9-4910-571-124P). 

0  The  lessons  should  require  students  to  check  battery 
connections  prior  to  using  STE/ICE. 

0  The  instructor's  CRT  does  not  provide  sufficient 
information. 


Another  respondent  felt  that  Segment  6  presented  appropriate  trouble¬ 
shooting  procedures,  but  did  not  present  underlying  troubleshooting  concepts. 

He  also  stated  that  the  illustrations  presented  on  the  student  CRT  in  Segment  4 
helped  students  to  learn  about  components  of  the  electrical  system. 

Trainee  Questionnaire.  Responses  of  the  ten  students  who  were  tested  on 
operational  equipment  are  reported  separately  from  the  responses  of  the  five 
students  who  were  not  tested  on  operational  equipment.  Students  who  were  not 
tested  participated  in  all  lessons  presented  by  the  simulator,  while  students 
who  were  tested  only  participated  in  a  subset  of  these  lessons  (Segments  0,  1, 
2,  3,  and  8). 


QUESTION  1;  How  do  you  feel  about  the  simulator? 

RATINGS:  1  «=  like  it  very  much,  5  •=  dislike  it  very  much 
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Response  Number  of  Responses 

Students  Tested  Students  Not  Tested 


(N  =  10)  (N  =  5) 


LIKE  IT  VERY  MUCH  6  1 
LIKE  IT  SOMEWHAT  3  0 
NO  STRONG  FEELINGS 

ABOUT  IT  0  1 
DISLIKE  IT  SOMEWHAT  0  0 
DISLIKE  IT  VERY  MUCH  J  _l 

TOTAL  10*  TOTAL  5 


The  mean  rating  for  students  who  were  tested  was  4.3  (standard  devia¬ 
tion  =  1.25).  The  mean  rating  for  students  who  were  not  tested  was  2.2 
(standard  deviation  =  1.79). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Many  students  who  were  tested  on  operational  equipment  gave  the  Grumman 
simulator  high  ratings  because  they  thought  the  device  was  a  good  training 
tool.  One  student  who  was  tested  disliked  the  simulator  very  much  because 
error  messages  appeared  on  the  student  CRT  when  correct  troubleshooting 
procedures  had  been  followed. 

In  general,  students  who  were  not  tested  on  operational  equipment  did 
not  rate  the  simulator  as  highly  as  students  who  were  tested.  Students  dis¬ 
liked  the  simulator  very  much  because  it  malfunctioned  frequently  and  because 
the  lessons  were  too  simple.  One  student  rated  the  simulator  highly  because 
he  felt  it  helped  him  understand  troubleshooting  procedures. 

QUESTION  3:  What  specific  features  of  the  simulator  do  you  like  or  dis¬ 
like? 


Students  who  were  tested  on  operational  equipment  liked  the  "hands-on" 
experience  with  the  3-D  module,  the  procedural! zed  instruction  presented  in 
the  lessons,  feedback  provided  to  students  about  the  correctness  of  their 
actions,  and  video  frames  that  identified  the  engine  components  to  which 
students  should  attend.  These  students  did  not  like  simulator  malfunctions, 
the  requirement  to  press  "next"  on  the  touch  screen  in  order  to  progress 
through  a  lesson,  wires  on  the  3-D  module  that  were  not  labeled,  and  lessons 
that  required  the  use  of  the  M110A2  howitzer  TM  and  the  STE/ICE  TM. 

One  student  who  was  not  tested  liked  the  audio  and  video  stimuli  pre¬ 
sented  on  the  student  monitor.  Another  student  liked  feedback  and  the  sim¬ 
plicity  of  the  device,  but  disliked  the  fact  that  students  could  input  data 
faster  than  the  device  could  accept  it.  Other  students  who  were  not  tested 
did  not  like  any  of  the  features  of  the  simulator. 

QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

Two  students  who  were  tested  and  who  liked  procedural ized  instruction 
felt  that  it  was  simple  to  follow  and  helped  them  to  identify  malfunctions. 
One  student  (tested)  liked  feedback  because  it  identified  student  errors  and 


reduced  training  time.  Another  student  (tested)  felt  that  unmarked  cables 
were  confusing  and  that  the  simulator  was  too  time-consuming.  No  other 
comments  about  specific  features  were  made  by  students  who  were  tested. 

Students  who  were  not  tested  reiterated  their  dislike  of  the  simulator 
because  it  malfunctioned  frequently.  One  student  disliked  the  lessons 
because  they  were  too  simple,  while  another  student  liked  the  information 
presented  on  the  student  monitor  because  it  was  simple. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

In  general,  students  who  were  tested  indicated  that  they  enjoyed  working 
with  the  simulator.  One  student,  however,  felt  that  the  Grumman  simulator 
was  worthless.  Another  student  stated  that  the  simulator  wasted  time  and 
money.  Other  students  stated  that  conventional  lecture  methods  were  as 
effective  as  the  simulator  and  that  the  simulator  should  be  drastically 
redesigned  before  it  is  accepted  by  the  Anny.  One  student  indicated  that 
students  should  receive  training  for  additional  tasks  on  the  simulator. 

Structured  Interview.  Responses  to  the  13  questions  that  were  posed 
during  the  structured  interview  are  presented  below. 

QUESTION  1:  What  were  the  instructional  features  of  the  simulator  that 
were  applicable  to  the  school's  training  course? 

Four  interviewees  felt  that  the  3-D  module  was  applicable  to  the  School's 
training  courso.  One  of  these  interviewees  noted  that  components  of  the  3-D 
module  looked  and  functioned  like  components  found  on  the  operational  equip¬ 
ment.  Another  interviewee,  however,  stated  that  the  STE/ICE  component  of  the 
3-D  module  did  not  perform  as  many  functions  as  an  operational  STE/ICE  kit. 

Most  interviewees  felt  that  the  video  feedback  presented  on  the  student 
CRT  was  applicable.  One  interviewee  indicated  that  the  feedback  allowed 
students  to  understand  what  they  were  doing,  while  another  interviewee  noted 
that  the  frequency  of  feedback  decreased  as  students  progressed  through 
lessons.  One  interviewee  criticized  feedback  because  of  the  delay  between 
students'  actions  and  the  arrival  of  feedback.  Another  interviewee 
criticized  the  feedback  because  students  were  not  always  provided  a  cue  to 
attend  to  the  student  CRT  when  feedback  was  presented.  This  interviewee 
stated  that  instructors  frequently  had  to  tell  students  to  attend  to  the  CRT 
in  order  to  receive  feedback. 

Three  interviewees  stated  that  audio  feedback  provided  to  students  was 
"good."  One  of  these  respondents  elaborated  on  his  comment  when  he  indicated 
that  audio  feedback  allowed  students  to  understand  the  results  of  their 
actions  without  having  to  read  the  student  CRT. 

Three  respondents  indicated  that  diesel  engine  sound  effects  were  appli¬ 
cable.  Two  of  these  respondents  stated  that  the  sound  effects  added  to  the 
realism  of  the  3-D  module.  One  respondent  noted  that  additional  sound  effects 
were  warranted  (an  engine  cranking,  but  not  starting). 

Four  interviewees  felt  that  the  hardcopy  printout  of  student  performance 
was  applicable  to  the  School's  training  course.  One  interviewee  indicated 
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that  this  feature  provided  instructors  the  capability  to  collect  data  on 
student  performance.  Another  interviewee,  however,  indicated  that  while 
the  printout  was  an  excellent  idea,  it  was  of  minimal  value  because  it  did 
not  identify  specific  errors  that  students  coinmitted. 

One  interviewee  stated  that  the  editing  capability  of  the  Grumman  device 
was  applicable;  however,  he  noted  that  none  of  the  instructors  who  operated 
the  device  were  trained  to  use  the  editing  system  (few  were  aware  of  its 
existence). 

Three  interviewees  felt  that  the  universal  instructor  feature  was 
applicable  because  it  allowed  instructors  to  access  different  lessons  in  the 
simulator's  curriculum.  Two  of  these  interviewees,  however,  felt  that  the 
utility  of  this  feature  was  diminished  because  a  printout  of  student  perform¬ 
ance  is  not  provided  when  this  feature  is  operational. 

Three  interviewees  felt  that  the  video  disc  system  was  applicable.  One 
of  these  individuals  felt  that  the  combination  of  still  and  motion  frames, 
computer-generated  text,  and  an  audio  track  were  effective  in  maintaining 
students'  motivation.  One  interviewee  criticized  this  feature  for  the 
length  of  time  required  to  search  for  frames.  Another  individual  stated  that 
the  student  CRT  was  too  small  and  that  instructors  had  to  assume  an  awkward 
position  in  order  to  turn  the  video  disc  player  on  and  off. 

Three  individuals  agreed  that  the  automated  pre-lesson  check  was  useful 
because  it  saved  time  for  instructors  and  guaranteed  that  the  device  was  in 
the  correct  configuration  to  present  a  lesson. 

The  "call  instructor"  feature  was  felt  to  be  applicable  by  two  inter¬ 
viewees.  One  interviewee  felt  that  this  feature  was  similar  to  the  "tilt" 
feature  of  a  pinball  game,  while  another  interviewee  indicated  that  this 
feature  helped  students  to  follow  correct  troubleshooting  procedures. 

Three  interviewees  felt  that  self-paced  lessons  were  applicable.  One  of 
these  interviewees  stated  that  this  was  one  of  the  better  features  of  the 
device.  These  same  three  individuals  also  felt  that  the  request  help  feature 
was  applicable. 

Ratings  made  by  interviewees  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 

whole. 

5 

3 

1 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

5 

4.4 

1.14 

QUESTION  2:  What  were  the  instructional  features  of  the  simulator  that 
were  not  applicable  to  the  school's  training  course? 

Two  respondents  noted  that  the  STE/ICE  manual  used  in  the  simulator 
curriculum  was  obsolete. 
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The  lessons  addressing  the  generator  and  starter  motor  were  seen  as 
inapplicable  by  two  interviewees  because  these  lessons  train  students  to 
perform  troubleshooting  procedures  that  are  not  used  in  conventional  training 
sessions  and  are  not  used  by  soldiers  in  the  field. 

Four  interviewees  felt  that  the  manner  in  which  the  lesson  were  arranged 
was  inappropriate.  Several  interviewees  stated  that  the  sequence  in  which 
lessons  were  presented  was  inflexible. 

One  interviewee  felt  that  the  instructor  CRT  was  inapplicable  because 
he  did  not  understand  the  information  that  was  presented  on  the  CRT  when 
students  were  participating  in  lessons.  Another  interviewee  indicated  that 
the  call  instructor  feature  was  not  useful  because  it  required  an  instructor 
to  spend  too  much  time  clearing  student  errors.  This  interviewee  felt  it 
would  be  more  appropriate  for  the  device  to  handle  all  errors  without 
requiring  instructor  intervention. 

QUESTION  3:  Which  features  of  the  lesson  presentation  helped  make  the 
simulator  lessons  interesting  to  the  students? 

Three  interviewees  felt  that  the  self-paced  nature  of  the  lessons  and 
the  request  help  feature  helped  to  maintain  student  interest.  One  inter¬ 
viewee  stated  that  these  features  allowed  students  to  ask  for  help,  something 
they  would  not  normally  do  during  conventional  training  sessions. 

Three  interviewees  stated  that  the  video  frames  presented  by  the  video 
disc  system  were  interesting  to  students,  while  two  interviewees  stated  that 
the  audio  track  that  accompanied  the  video  presentation  was  of  interest  to 
students.  The  video  presentation  was  fun  to  watch,  while  the  audio  presenta¬ 
tion  allowed  students  to  learn  troubleshooting  procedures  without  requiring 
them  to  read. 

Other  features  mentioned  by  interviewees  included  feedback  to  students, 
touch  panel,  request  help,  and  the  3-D  module. 

Ratings  made  by  interviewees  are  presented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

5 

3.6 

1.14 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

5 

5.4 

.55 

QUESTION  4:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  particularly  effective? 

Three  interviewees  felt  that  the  introduction  to  AMTESS  and  Lesson  1 
(set-up  and  check-out  of  the  STE/ICE  kit)  were  effective.  One  interviewee 
indicated  that  parts  of  all  three  lessons  were  effective  because  they  pro¬ 
vided  students  hands-on  experience  with  the  3-D  module. 
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Ratings  made  by  interviewees  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  ■  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  lessons  as  a 
whole. 

5 

3.2 

.84 

Rate  the  potential 
training  value  of  these 
lessons  as  a  whole. 

5 

4.8 

1.1 

QUESTION  5:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  particularly  ineffective? 

Interviewees  generally  agreed  that  the  lessons  that  addressed  trouble¬ 
shooting  the  starting  and  charging  system  (Segments  4,  5,  6,  7,  8,  and  9) 
were  ineffective.  Three  interviewees  felt  that  Segments  6  and  9  were  parti¬ 
cularly  ineffective  because  these  segments  present  troubleshooting  pro¬ 
cedures  that  students  cannot  use  in  the  field. 

One  interviewee  commented  that  the  lessons  should  have  trained  students 
to  use  a  multimeter  rather  than  the  STE/ICE  kit.  This  interviewee  also  stated 
that  the  request  help  feature  allowed  students  to  complete  troubleshooting 
lessons  too  easily  since  students  could  request  correct  procedures  from  the 
simulator  rather  than  trying  to  complete  exercises  on  their  own.  One  inter¬ 
viewee  felt  that  the  first  two  segments  were  ineffective  because  they  were 
advertisements  for  AMTESS. 

QUESTION  6:  Were  there  any  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  being  presented  by  the  simulator  was  above  the  students? 

None  of  the  interviewees  felt  that  material  presented  by  the  simulator 
was  too  difficult  for  students. 

QUESTION  7:  Were  there  any  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  presented  by  the  simulator  should  be  increased? 

All  of  the  interviewees  identified  instances  in  which  the  difficulty 
level  of  material  presented  to  students  should  be  increased.  Generally, 
interviewees  felt  that  the  rate  at  which  material  was  presented  to  students 
was  too  slow.  (Although  lessons  were  self-paced,  all  students  were  required 
to  complete  each  step  of  a  lesson.  Further,  the  rate  at  which  students  per¬ 
formed  procedures  on  the  3-D  module  was  frequently  limited  by  the  simulator's 
ability  to  process  student  input.)  Three  interviewees  felt  that  the  lessons 
did  not  address  the  full  capability  of  the  STE/ICE  kit.  All  five  interviewees 
felt  that  the  lessons  should  have  placed  more  emphasis  on  the  use  of  TMs 
(especially  the  vehicle  operator's  manual  for  starting  and  stopping  the 
engine).  Three  interviewees  felt  that  students  should  have  experienced 
troubleshooting  more  difficult  problems  on  the  3-D  module,  while  two  inter¬ 
viewees  felt  that  the  difficulty  level  of  the  troubleshooting  procedures  was 
adequate.  One  interviewee  felt  that  the  lessons  should  have  addressed  the 
deductive  reasoning  process  involved  in  t*"oubleshooting  instead  of  simply 
presenting  troubleshooting  procedures. 
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QUESTION  8;  What  hardware  features  of  the  simulator  made  it  more 
effective  than  conventional  training? 

Statements  made  by  four  of  the  interviewees  indicated  that  they  felt 
that  the  3-D  module  was  of  value  because  it  provided  easy  access  to  components, 
it  did  not  pose  a  safety  hazard  to  students,  and  it  could  not  be  damaged  as 
easily  as  operational  equipment.  One  interviewee  disliked  the  fact  that  the 
3-D  module  provided  students  with  access  to  components  that  were  not  easily 
accessible  on  operational  equipment.  Other  features  mentioned  by  interviewees 
include  the  video  disc  system,  the  request  help  feature,  and  the  touch  panel. 

Ratings  made  by  interviewees  are  presented  below: 


RATING'; 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 


whole. 

5 

3.6 

.55 

Rate  the  potential 

training  value  of  these 

features  as  a  whole. 

5 

4.4 

.89 

QUESTION  9:  What  were  the  features  that  made  this  simulator  easy  for 
the  instructors  to  operate? 

Several  interviewees  stated  that  the  touch  panel  and  the  automated  pre 
lesson  check  of  the  3-D  module  made  the  simulator  easy  to  operate.  Inter¬ 
viewees  stated  that  they  did  not  understand  some  features  of  the  simulator 
(editing  system,  instructor  CRT)  and  they  also  felt  that  the  device  was 
difficult  to  operate  when  certain  other  features  malfunctioned  (3-D  module, 
video  disc  system)  or  provided  useless  information  (instructor  CRT). 

Ratings  made  by  interviewees  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

5 

3.4 

.55 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

5 

4.6 

.89 

QUESTION  10:  What  were  the  features  that  made  this  simulator  easy  for 
the  students  to  operate? 

The  interviewees  generally  agreed  that  the  touch  panel  enabled  students 
to  enter  responses  into  the  device  quickly  and  easily.  One  interviewee,  how¬ 
ever,  noted  that  the  touch  panel  did  not  always  record  students'  responses 
correctly. 


An  of  the  interviewees  indicated  that  the  audio  track  of  the  video  disc 
player  helped  to  make  the  device  easy  for  students  to  use.  Several  inter¬ 
viewees  felt  that  students  were  able  to  process  auditory  stimuli  much  better 
than  written  stimuli. 

Three  interviewees  felt  that  the  acce‘‘:  lity  of  components  on  the  3-D 
module  made  the  device  easy  to  operate,  wh.'  le  interviewee  felt  that  such 
accessibility  confused  students.  One  individual  noted  that  students  became 
frustrated  when  the  3-D  module  malfunctioned. 

Ratings  made  by  interviewees  are  presented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

5 

4.2 

.45 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

5 

4.4 

.89 

QUESTION  11:  What  aspects  of  the  device  would  be  appropriate  for 
substituting  for  actual  equipment? 

Interviewees  experienced  difficulty  answering  the  question  and  appeared 
to  reinterpret  the  question  as  "What  aspects  of  the  device  made  it  appropriate 
for  substituting  for  actual  equipment?".  Four  interviewees  stated  that  the 
3-D  module  made  the  device  appropriate  for  substitution  for  operational  equip¬ 
ment.  They  noted  that  malfunctions  are  easily  inserted  into  the  3-D  module, 
the  module  is  safer  than  operational  equipment,  and  the  probability  of 
damaging  the  3-D  module  is  low.  These  same  four  interviewees  also  indicated 
that  presenting  self-paced  lessons  with  a  video  disc  player  was  valuable 
because  students  were  able  to  view  detailed  video  frames  at  a  rate  that  they 
selected. 


Ratings  made  by  interviewees  are  presented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

5 

3 

1 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

5 

3.8 

.84 

QUESTION  12:  What  types  of  problems 

did  't-udents 

have? 
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All  five  interviewees  indicated  that  there  were  frequent  hardware  and 
software  problems  with  the  device.  These  problems  are  summarized  below: 

0  Simulator  recorded  correct  student  actions  as  errors 
or  recorded  student  errors  as  correct  actions. 

0  System  would  "lock-up,"  i.e.,  would  not  record 
student  responses  or  advance  to  the  next  lesson. 

0  System  "lock-ups"  required  students  to  repeat 
lessons  that  were  in  p-'oqress  when  the  failure 
occurred. 

0  Simulator  failed  to  record  the  responses  of  students 
for  lessons  that  had  been  completed. 

0  Cables  and  connectors  for  the  3-D  module  shorted  out. 

0  Components  of  the  3-D  module  could  not  withstand 
normal  student  use. 

0  Material  presented  by  the  video  disc  player  was  not 
synchronized  with  events  that  occurred  on  the  3-D 
module. 

Other  problems  mentioned  by  the  interviewees  are  summarized  below: 

0  The  student  CRT  was  too  small  to  be  seen  by  more 
than  one  student  at  a  time. 

0  Students  found  it  difficult  to  look  down  at  the 
student  CRT  when  they  were  working  with  the  3-D 
module  in  a  standing  position. 

0  Students  did  not  know  if  they  should  attend  to 
the  3-D  module,  the  student  CRT,  or  a  TM. 

0  Students  could  not  review  the  frame  that  preceded 
the  current  frame  presented  by  the  video  disc 
player. 

0  Many  students  entered  responses  at  a  rate  that 
was  faster  than  the  system  could  accommodate, 
causing  students  to  wait  for  the  system  to 
"catch  up"  with  them. 

0  The  student  guidebook  was  difficult  to  read  and 
the  information  it  provided  was  elementary. 

QUESTION  13:  How  would  you  employ  the  simulator  in  order  to  gain  maxi 
mum  benefit  from  it? 

All  of  the  interviewees  felt  that  the  device  should  be  used  to  train 
entry  level  students  because  the  tasks  addressed  by  the  simulator  are 
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relatively  simple  and  because  the  material  is  presented  at  a  slow  rate.  In 
addition,  most  of  the  interviewees  stated  that  training  with  the  simulator 
should  be  supplemented  by  training  with  operational  equipment  in  order  to 
familiarize  students  with  the  location  of  components  on  operational  equipment. 

Ratings  made  by  the  interviewees  for  the  simulator  as  a  whole  are  pre¬ 
sented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  the  simulator  as  a 
whole. 

5 

3 

1 

Rate  the  potential 
training  value  of  the 
simulator  as  a  whole. 

5 

4.2 

.84 

Data  Collector  Observations.  The  observations  made  by  the  data  collector 
during  the  transfer-of-training  study  are  summarized  below: 

0  The  system  malfunctioned  frequently.  System  failures 
resulted  in  a  loss  of  student  data  and  a  decrease  in 
student  and  instructor  desire  to  work  with  the  device. 

0  Segments  6  and  9  are  of  little  value  to  63H30  students. 

In  these  segments,  students  are  trained  to  connect  leads 
directly  to  the  generator  and  starter  motor.  However, 
on  operational  equipment,  this  procedure  is  rarely  per¬ 
formed  because  it  requires  the  engine  to  be  removed  from 
the  vehicle.  Conventionally  trained  students  learn  to 
identify  a  faulty  generator  or  starter  motor  through 
deductive  reasoning,  a  process  which  does  not  require 
the  engine  to  be  removed  from  the  vehicle. 

0  Few  individuals  were  aware  of  the  existence  of  an  editing 
system.  Those  familiar  with  the  system  found  it  awkward 
and  time  consuming  because  it  was  too  heavily  oriented 
towards  inflexible,  repetitive  menus.  Greater  emphasis 
should  be  placed  on  direct  edit  commands. 

0  The  record  of  student  performance  is  of  little  value 
since  it  does  not  identify  specific  actions  the  student 
performs.  Total  time  required  to  complete  a  lesson, 
total  number  of  errors  committed,  and  an  overall  per¬ 
formance  index  are  the  only  values  that  appear  on  the 
printout  for  each  lesson. 

0  When  students  make  two  consecutive  errors,  a  "call  your 
instructor"  message  appeared  on  the  student  CRT.  If  the 
image  that  was  on  the  CRT  immediately  prior  to  these 
errors  was  computer-generated  text,  then  the  CRT  became 
blank  after  the  student  errors  were  cleared.  Since 


instructors  were  given  no  cue  about  the  next  step 
to  perform,  they  spent  considerable  amounts  of  time 
attempting  to  find  the  next  appropriate  step.  Fre¬ 
quently,  the  correct  step  could  not  be  identified 
and  the  segment  would  have  to  be  started  from  the 
beginning,  resulting  in  a  loss  of  student  data. 

0  An  audio  cue  (chirp)  emitted  from  the  3-D  module 
indicated  that  students  should  look  at  the  CRT. 

One  of  two  types  of  messages  appeared  on  the  CRT 
in  conjunction  with  the  audio  cue,  a  message 
indicating  that  an  error  was  committed  or  a 
message  indicating  that  a  procedure  should  be  per¬ 
formed.  Students  were  frequently  confused  because 
they  assumed  that  the  audio  cue  could  only  mean 
that  they  had  committed  an  error.  The  use  of  two 
different  cues  seems  warranted. 

0  Students  frequently  performed  troubleshooting 
procedures  on  the  3-D  module  faster  than  the  rate 
at  which  the  simulator  could  record  student  input. 
Consequently,  students  were  provided  error  messages 
when,  in  fact,  they  had  not  committed  any  errors. 

Discussion 


The  preceding  section  presented  a  wide  range  of  opinions  about  various 
aspects  of  the  Grumman  training  device.  Several  common  themes  that  appeared 
throughout  the  data  are  summarized  below  in  the  form  of  positive  and  negative 
statements  about  the  device. 

Positive  Statements.  The  ability  to  perform  troubleshooting  tasks  on 
the  3-D  module  is  a  valuable  device  feature.  Students  stated  that  they 
enjoyed  working  on  the  3-D  module.  Instructors  and  course  developers  felt 
that  "hands-on"  experience  with  the  module  helped  students  understand  trouble¬ 
shooting  procedures. 

The  video  disc  system  is  an  effective  and  motivating  device  feature. 
Students  stated  that  they  enjoyed  the  audio  and  video  material  presented  on 
the  video  disc  system.  Instructors  felt  that  material  presented  on  this 
system  helped  students  to  identify  and  troubleshoot  engine  components. 

Negative  Statements.  The  Grumman  device  frequently  malfunctions.  A1 1 
of  the  individuals  who  provided  questionnaire  data  or  interview  data  felt 
that  the  effectiveness  of  the  Grumman  device  was  diminished  because  of  its 
low  reliability.  The  device  exhibited  many  different  types  of  malfunctions 
that  caused  numerous  delays  in  training  sessions. 

Lessons  are  inflexible.  Students  did  not  like  the  fact  that  they  could 
not  skip  segments  (or  parts  of  segments),  nor  could  they  repeat  segments  (or 
parts  of  segments)  on  request.  Instructors  did  not  like  the  fact  that  they 
could  not  manipulate  the  order  in  which  segments  are  presented  to  students. 
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Some  lessons  are  too  simple  while  others  are  inappropriate.  Students, 
instructors,  and  course  developers  commented  that  the  lessons  were  more 
appropriate  for  entry  level  students  than  they  were  for  advanced  students 
(63H30)  because  they  were  relatively  simple.  Instructors  and  course  developers 
stated  that  Segments  6  and  9  were  inappropriate  because  they  present  trouble¬ 
shooting  procedures  that  cannot  be  used  by  soldiers  in  the  field. 

The  student  performance  record  is  of  little  value.  Although  many  indi- 
viduals  thought  that  a  printout  of  student  performance  was  conceptually 
appropriate,  they  felt  that  the  information  presented  by  the  Grumman  device 
was  inadequate  because  it  was  not  presented  in  sufficient  detail. 

System  response  time  is  too  slow.  Both  students  and  instructors  com¬ 
mented  that  the  simulator  could  not  accept  and  process  student  input  at  a 
rate  of  speed  that  was  commensurate  with  typical  student  performance.  This 
resulted  in  training  delays  and  a  decreased  desire  (by  students  and  instruc¬ 
tors)  to  work  with  the  device. 


Experiment  4 

Twenty-three  (23)  students  from  the  63D30  MOS  (Self-propelled  Field 
Artillery  Systems  Mechanic)  were  trained  to  perform  a  task  (identical  to  the 
task  in  Experiment  3)  using  either  conventional  methods  (11  students)  or  the 
Grumman  simulator  (12  students).  Students  were  then  tested  on  their  ability 
to  perform  this  task  on  operational  equipment.  Data  consist  of  responses  to 
the  trainee  questionnaire  only.  Individuals  responsible  for  training  MOS 
63030  students  were  also  responsible  for  training  MOS  63H30  students.  Indi¬ 
viduals  responsible  for  creating  and  modifying  curricula  for  the  63030  MOS 
were  also  responsible  for  the  63H30  MOS.  Since  data  obtained  from  these 
individuals  has  already  been  reported  in  Experiment  3;  it  will  not  be 
reported  in  this  experiment. 

Method 


Subjects.  Characteristics  of  subjects  who  provided  data  are  as  follows: 

0  Trainee  questionnaire.  Twelve  (12)  63030  students  who 
were  trained  with  the  Grumman  simulator  completed  this 
questionnaire  after  they  were  tested  on  their  ability 
to  perform  a  troubleshooting  task  on  operational  equip¬ 
ment.  The  questionnaire  was  also  completed  by  three 
students  who  were  trained  with  the  simulator,  but  were 
not  tested  on  their  ability  to  perform  a  troubleshooting 
task  on  operational  equipment. 

Materials.  The  trainee  questionnaire  used  in  this  experiment  was 
identical  to  the  questionnaire  used  in  previous  experiments.  (See 
Appendix  A.) 

Procedure.  The  procedure  used  in  administering  the  trainee  questionnaire 
was  highly  similar  to  the  procedure  used  in  previous  experiments.  Trainees 
responded  to  the  questionnaire  after  training  sessions  were  completed  (for 
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trainees  who  were  not  subsequently  tested  on  operational  equipment)  or  after 
training  and  testing  were  completed.  Trainees  were  briefed  before  responding 
to  the  questionnaire. 

Responses  of  the  12  students  who  were  tested  on  operational  equipment 
are  reported  separately  from  responses  of  the  three  students  who  were  not 
tested  on  operational  equipment. 

Results 


QUESTION  1:  How  do  you  feel  about  the  simulator? 

RATINGS:  1  =  dislike  it  very  much,  5  =  like  it  very  much 


Response 


LIKE  IT  VERY  MUCH 
LIKE  IT  SOMEWHAT 
NO  STRONG  FEELINGS 
ABOUT  IT 

DISLIKE  IT  SOMEWHAT 
DISLIKE  IT  VERY  MUCH 


Number  of  Responses 

Students  Tested 
(N  =  12) 

5 
3 


Students  Not  Tested 
(N  =  3) 

2 

0 


2 

1 

1 

TOTAL  T2 


1 

0 

_0 

TOTAL  3 


The  mean  rating  for  students  who  were  tested  was  3.83  (standard  devia¬ 
tion  =  1.34).  The  mean  rating  for  students  who  were  not  tested  was  4.33 
(standard  deviation  =  1.15). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Several  students  who  were  tested  and  who  gave  the  device  a  high  rating 
liked  the  feedback  provided  by  the  simulator.  Other  students  who  were 
tested  stated  that  the  device  would  be  a  useful  training  tool  for  entry  level 
students. 


One  of  the  students  who  was  not  tested  gave  the  simulator  a  high  rating 
because  simulator  lessons  made  it  easy  for  him  to  learn  to  use  STE/ICE. 
Another  student  who  was  not  tested  gave  the  simulator  a  neutral  rating 
because  it  did  not  function  properly. 

QUESTION  3:  What  specific  features  of  the  simulator  do  you  like  or  dis¬ 
like? 


Students  who  were  tested  liked  the  fidelity  of  the  3-D  module,  the  video 
disc  system,  the  request  help  feature,  feedback,  the  requirement  to  think 
about  troubleshooting  procedures,  the  absence  of  an  instructor,  and  pro¬ 
cedural  i  zed  self-paced  lessons.  One  of  these  students  stated  that  the  device 
recorded  correct  student  actions  as  errors  and  that  the  device  operated  too 
slowly. 

None  of  the  students  who  were  not  tested  made  specific  comments  about 
features  they  liked  or  disliked. 
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QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

One  student  who  was  tested  liked  procedural i zed  self-paced  lessons 
because  they  make  it  easy  to  learn.  No  other  students  made  relevant 
responses. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

Students  who  were  tested  made  the  following  comments:  1)  the  simulator 
is  not  as  good  as  conventional  training;  2)  the  simulator  is  preferable  to 
an  instructor;  3)  students  should  be  trained  to  use  STE/ICE  before  using 
the  simulator;  and  4)  the  simulator  should  be  used  by  one  student  at  a  time. 
Students  who  were  not  tested  did  not  respond  to  this  question. 

Discussion 


Only  data  collected  from  63D30  trainees  has  been  reported  in  this  experi¬ 
ment  (other  data  concerning  the  Grumman  device  at  APG  is  presented  in  Experi¬ 
ment  3).  This  data  can  be  summarized  by  one  positive  and  one  negative 
statement. 

Positive  Statement.  Students  liked  the  simulator  and  a  number  of 
specific  features.  §]^ci fic  features  that  students  liked  include  feedback , 
lessons  addressing  STE/ICE,  the  3-D  module,  the  request  help  feature,  the 
video  disc  system,  and  procedural i zed  lessons. 

Negative  Statement.  The  simulator  frequently  malfunctioned  and 
operated  too  slowlj^  These  comments  were  also  made  ty  63H30  students 
(Experiment  3). 


Experiment  5 

Twenty-two  (22)  students  from  the  24C10  MOS  (Hawk  Missile  Firing 
Section  Mechanic)  were  trained  to  perform  troubleshooting  tasks  using  either 
convent''onal  methods  (12  students)  or  the  Seville/Burtek  simulator  (10 
students).  These  students  were  then  tested  on  their  ability  to  perform 
troubleshooting  tasks  (identification  of  faulty  components)  on  operational 
equipment.  Data  consist  of  responses  to  questionnaires  and  structured  inter¬ 
views,  as  well  as  observations  made  by  the  on-site  data  collector  during  the 
course  of  the  experiment. 

Method 


Subjects.  Fewer  subjects  were  available  to  complete  questionnaires  and 
participate  in  interviews  in  Experiment  5  than  in  previous  experiments. 
Characteristics  of  subjects  who  provided  data  are  as  follows: 

0  Initial  instructor  questionnaire.  Four  civilian  and  seven 
military  instructors  completed  the  questionnaire. 

0  Instructor  questionnaire.  One  of  the  civilian  instructors 
described  above  completed  the  instructor  questionnaire. 
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0  Course  developer  questionnaire.  No  course  developers 
were  available  to  complete  this  questionnaire. 

0  Trainee  questionnaire.  Ten  MOS  24C10  students  who 
were  trained  with  the  Seville/Burtek  device  completed 
this  questionnaire  after  they  were  tested  on  their 
ability  to  perform  troubleshooting  tasks  on  opera¬ 
tional  equipment. 

0  Structured  interview.  The  civilian  instructor 
described  above  and  the  on-site  data  collector  for 
the  transfer-of-training  study  completed  structured 
interviews. 

Materials.  The  questionnaires  and  structure  interview  form  used  in 
Experiment  5  were  identical  to  those  used  in  the  previous  experiments. 

(See  Appendix  A.) 

Procedure.  The  procedure  followed  in  administering  the  questionnaires 
and  interviews  was  highly  similar  to  the  procedure  described  for  previous 
experiments. 

Respondents  commented  on  specific  features  of  the  Seville/Burtek 
device.  These  features  are  described  in  Table  6. 

Results 


Initial  Instructor  Questionnaire.  Responses  to  this  questionnaire  are 
summarized  below. 

QUESTION  1:  How  easy  or  hard  is  it  to  learn  to  use  the  simulator? 
RATINGS:  1  =  very  hard,  5  =  very  easy) 

N  =  11 


Response 


Number  of  Responses 


VERY  EASY  6 
FAIRLY  EASY  0 
AVERAGE  1 
FAIRLY  HARD  4 
VERY  HARD  D 


TOTAL  TT 


The  mean  rating  for  all  instructors  was  3.73  (standard  deviation  =  1.49). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Instructors  who  gave  the  simulator  neutral  and  high  ratings  agreed  that 
the  training  course  conducted  by  Seville/Burtek  was  valuable  and  that  the 
written  instructions  for  operating  the  simulator  were  clear  and  easy  to 
understand.  These  instructors  also  stated  that  their  previous  experience 
with  the  Hawk  radar  unit  and  with  computers  helped  them  understand  how  to 
operate  the  simulator.  The  instructors  who  stated  that  it  was  fairly  hard 
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to  learn  to  use  the  simulator  said  that  there  were  too  many  problems  with 
the  simulator.  They  stated  that  they  did  not  like  learning  to  use  a  simula¬ 
tor  that  was  not  operating  correctly. 

QUESTION  3:  How  do  you  feel  about  the  simulator? 
fWTINGS:  1  »  dislike  it  very  much,  5  *=  like  it  very  much 
N  =  11 


Response  Number  of  Responses 

LIKE  IT  VERY  MUCH  1 

LIKE  IT  SOMEWHAT  3 

NO  STRONG  FEELINGS  ABOUT  IT  0 

DISLIKE  IT  SOMEWHAT  6 

DISLIKE  IT  VERY  MUCH  1 

TOTAL  TT 

The  mean  rating  for  all  instructors  was  2.73  (standard  deviation  =  1.27). 

QUESTION  4:  Why  did  you  answer  Question  3  as  you  did? 

The  instructors  who  liked  the  simulator  indicated  it  provided  them  the 
opportunity  to  train  tasks  (high  voltage  procedure)  that  were  impossible  to 
train  on  operational  equipment.  In  addition,  these  instructors  felt  that 
the  simulator  was  more  reliable  than  the  operational  equipment  (which  fre¬ 
quently  malfunctions).  Instructors  who  disliked  the  simulator  indicated 
that  they  did  not  trust  the  device  because  it  was  not  operating  properly. 

(The  simulator  malfunctioned  frequently  for  this  group  of  instructors.) 

These  individuals  disliked  the  simulator  because  it  was  not  realistic.  They 
noted  that  components  were  missing  or  not  true  to  life  (i.e.,  degraded 
fidelity  of  the  cabinet,  components,  switches,  and  dials),  and  that  trouble¬ 
shooting  procedures  presented  by  the  simulator  were  different  (in  an  unspeci¬ 
fied  manner)  from  the  procedures  normally  presented  to  students. 

QUESTION  5;  Please  make  any  additional  comments  you  feel  are  appropriate. 

Several  respondents  noted  that  the  cost  of  the  simulator  was  too  high, 
and  that  its  use  could  not  justify  its  cost  since  the  Hawk  radar  was 
scheduled  for  significant  hardware  and  software  modifications  in  the  near 
future.  (Since  no  accurate  cost  projections  for  the  device  were  available, 
these  comments  were  apparently  based  on  rumor.) 

Instructor  Questionnaire.  Data  reported  for  the  instructor  questionnaire 
are  based  on  the  responses  of  a  single  instructor.  Rather  than  completing  a 
questionnaire  for  each  of  the  50  exercises  taught  by  the  simulator  (see  Table 
6),  the  instructor  grouped  the  exercises  and  completed  a  questionnaire  for 
each  of  the  following  six  content  areas: 

1.  introduction  to  the  transmitter 

2.  master  oscillator  and  power  amplifier  high 
voltage  circuits 

3.  modulator  bias  and  arc  detection  circuits 
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Table  6 


Exercises  Presented  by  the  Seville/Burtek  Simulator 
in  the  Radar  Transmitter  Configuration 


Exercise  Number 


Description 


1 

2 

3 

4 

s 

6 

7 

B 

9 

10 
11 
12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 
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Nonnal  Operations 
Table  3*16 
Monthly  Check 

Remove  and  Replace  High  Voltage  Regulator 
Remove  and  Replace  Master  Oscillator  (HO)  Filament 
PoMcr  Supply  (PS)  A4 

Remove  and  Replace  Power  Amplifier- (PA)  Filament 
PS  A3 

Remove  and  Replace  MO 

Remove  and  Replace  PA  Tube 

Remove  and  Replace  HI/LO  Frequency  Amplifier 

Adjust  MO  Frequency 

MO  Filament  Test 

PA  Filament  Test 

PA  High  Voltage  Test 

MO  High  Voltage  Test 

Degeneration  Intermediate  Frequency  (IF)  Amplifier 
Test  » 

High  Voltage  Regulator  (PA)  Failure  [Ar 
High  Voltage  Regulator  (PA)  Failure  ,8  * 

High  Voltage  Regulator  (PA)  Failure  ,C|“ 

High  Voltage  Regulator  (PA)  Failure  [Dj 
MO  High  Voltage  PS  A1  Failure  [A] 

MO  High  Voltage  PA  A1  Failure  [DJ 
PA  PS  A2  Failure  [A] 

PA  PS  A2  Failure  [DJ 
PA  Filament  PS  A3  Failure  [A' 

PA  Filament  PS  A3  Failure  ,D, 

PA  Filament  PS  A3  Failure  D 
MO  Filament  PS  A4  Failure  A 
MO  Filament  PS  A4  Failure  8 
MO  Filament  PS  A4  Failure  C 
MO  Filament  PS  A4  Failure  [D, 

PA  Tube  Failure  [A] 

PA  Tube  Failure  [DJ 
MO  Failure  [A] 

MO  Failure  [EJ* 

Ferrite  Switch  Failure  [A] 

Ion  Probe  Failure  [A] 

Ferrite  Switch  Control  and  PS  Failure  [A] 

Ferrite  Switch  Control  and  PS  Failure  [DJ 
Range  and  Coding  Amplifier  Oscillator  Failure  [A] 
Range  and  Coding  Amplifier  Oscillator  Failure  [OJ 
Monitor  Amplifier  Failure  [A] 

Degeneration  IF  Amplifier  Failure  [A] 

Degeneration  IF  Amplifier  Failure  [D] 

Bridge  Null  and  Cavity  Tuning  Control  Amplifier 
Failure  [A] 

Bridge  Null  and  Cavity  Tuning  Control  Amplifier 
Failure  [D] 

Ferrite  Phase  Controller  (Degen)  Failure  [A' 

High  and  Low  Frequency  Amplifier  Failure  A, 

High  and  Low  Frequency  Amplifier  Failure  [6. 
Klystron  Tuning  Control  Amplifier  [A] 

Weekly  Check 


“[A],  [8],  [C],  [D],  and  [E]  denote  exercise  variations. 


i! 


I 


K 


V 


F: 


h 


LV 


K 


4.  noise  degeneration  circuits 

5.  radio  frequency  generation  circuits 

6.  modulation  circuits 

QUESTION  1:  How  well  does  the  simulator  train  this  task? 
MTINGS:  1  *  very  poorly,  5  =  very  well 
N  =  1 


Response 


Number  of  Responses 


VERY  WELL  3 
MODERATELY  WELL  2 
FAIRLY  0 
POORLY  1 
VERY  POORLY  0 


TOTAL  I 


The  mean  rating  for  all  tasks  was  4.17  (standard  deviation  ■  1.17). 

Raw  data  are  presented  in  Table  E-1. 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Certain  tasks  were  given  the  rating  "very  well"  because  these  tasks  could 
not  be  trained  using  operational  equipment.  The  instructor  did  not  indicate 
why  some  tasks  were  given  a  rating  of  "moderately  well"  or  "poorly." 

QUESTION  3:  What  specific  features  of  the  simulator  (involved  in  this 
task  do  you  like  or  dislike? 


Specific  features  of  the  simulator  that  appealed  to  the  instructor  were: 
1)  the  overall  high  fidelity  of  the  3-D  module,  2)  the  ability  to  insert  mal¬ 
functions  without  damaging  equipment,  and  3)  the  ability  to  demonstrate  the 
built-in  test  equipment.  The  instructor  disliked:  1)  the  low  fidelity  of 
certain  (unspecified)  components  of  the  3-D  module,  and  2)  the  cable  connec¬ 
tors. 


QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

The  instructor  liked  the  3-D  module  because  it  allowed  him  to  demon¬ 
strate  concepts  and  normal  operations  of  the  transmitter  without  the  distrac¬ 
tion  and  complexity  present  in  the  operational  equipment.  He  found  the  sim¬ 
ulator  to  be  easy  to  set  up,  reliable,  and  friendly  to  both  instructors  and 
students. 

The  instructor  indicated  that  the  ability  to  train  high  voltage  pro¬ 
cedures,  radio  frequency  generation  tasks,  and  modulation  bias  and  arc 
detector  tasks  was  the  outstanding  capability  of  the  Seville/Burtek  device. 
He  noted  that  malfunctions  in  these  circuits  could  be  inserted  into  the  simu 
lator  but  not  the  operational  equipment  because  the  malfunction  would  damage 
the  operational  equipment. 
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The  instructor  noted  that  simulator-trained  students  are  trained  to  use 
the  high  voltage  test  set  to  isolate  complex  problems.  However,  students 
trained  on  operational  equipment  are  not  taught  to  use  this  equipment  since 
insertion  of  high  voltage  malfunctions  damages  the  equipment. 

The  instructor  noted  that  the  simulator  provided  poor  training  for  the 
noise  degeneration  circuits.  He  noted  that  since  many  of  the  details  of 
these  circuits  were  not  represented  in  the  simulator,  students  experienced 
difficulty  troubleshooting  these  circuits  on  the  operational  equipment.  The 
instructor  disliked  the  cable  connectors  since  they  were  difficult  to  use. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

The  instructor  indicated  that  the  simulator  was  superior  to  conventional 
training  methods  for  five  of  the  six  content  areas  (see  page  54)  addressed  by 
the  simulator.  (Noise  degeneration  circuits  were  trained  poorly.)  He  stated 
that  he  could  depend  on  the  simulator  to  function  correctly. 

Trainee  Questionnaire.  Responses  of  the  10  students  who  were  trained 
with  the  simulator  are  summarized  below. 

QUESTION  1:  How  do  you  feel  about  the  simulator? 

MTINGS:  1  ■  dislike  it  very  much,  5  =  like  it  very  much 

N  =  10 


Response  Number  of  Responses 

LIKE  IT  VERY  MUCH  5 

LIKE  IT  SOMEWHAT  5 

NO  STRONG  FEELINGS  ABOUT  IT  0 

DISLIKE  IT  SOMEWHAT  0 

DISLIKE  IT  VERY  MUCH  0 

TOTAL  W 

The  mean  rating  for  all  students  was  4.5  (standard  deviation  «=  .53). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Students  indicated  that  they  were  comfortable  and  confident  working  on 
operational  equipment  because  the  simulator  helped  them  overcome  their  fear 
of  the  operational  equipment.  They  also  felt  that  the  self-paced  lessons 
helped  them  to  learn  troubleshooting  procedures. 

QUESTION  3:  What  specific  features  of  the  simulator  do  you  like  or  dis¬ 
like? 


Students  liked  the  variety  of  malfunctions  that  could  be  inserted  into 
the  simulator,  the  lack  of  noise  generated  by  the  cooling  system  of  operational 
equipment,  feedback,  the  self-paced  lesson  presentation,  and  the  hardcopy 
printout.  Students  disliked  the  low  fidelity  of  certain  3-D  module  components 
and  the  lack  of  time  spent  training  on  the  operational  equipment. 

QUESTION  4:  Why  do  you  like  or  dislike  these  features? 


57 


Students  liked  troubleshooting  realistic  problems  and  symptoms  that 
could  not  be  placed  into  the  operational  equipment  (e.g.,  high  voltage 
problems).  They  felt  prepared  to  troubleshoot  these  types  of  problems  in 
the  field  after  they  were  trained  on  the  simulator.  The  lack  of  usual 
radar  noise  helped  students  to  hear  the  instructor  clearly.  Several  stated 
that  feedback  provided  by  the  simulator  was  useful,  while  one  felt  that  feed¬ 
back  should  be  reduced  as  students  become  more  proficient  troubleshooters. 

All  students  preferred  self-paced  training  and  the  last  good-first  bad 
method  of  troubleshooting  presented  in  the  simulator  lessons,  rather  than 
lockstep  training  and  Fault  Isolation  Procedures  (FIP)  used  in  conventional 
training. 

The  hardcopy  printout  was  helpful  since  the  printout  enabled  them  to 
see  which  of  their  actions  were  correct  and  which  were  incorrect.  They  dis¬ 
liked  the  low  fidelity  of  the  3-D  module  (tuning  motors,  crystals,  switches, 
and  meters  were  missing  or  were  low  fidelity  representations  of  the  opera¬ 
tional  equipment).  It  was  difficult  to  understand  how  certain  circuits 
worked  since  components  of  these  circuits  were  not  represented  in  the  simula¬ 
tor.  AUo  it  was  difficult  to  remember  to  check  certain  meters  on  the  opera¬ 
tional  equipment  since  the  simulator's  low  fidelity  meter  always  indicated 
that  the  meter  was  providing  a  "good"  reading. 

QUESTION  5:  Please  make  aqy  additional  comments  you  feel  are  appropriate. 

Many  students  indicated  that  training  could  have  been  improved  if  more 
time  had  been  provided  for  practice  on  the  operational  equipment  since  certain 
tasks  (remove/replace  certain  components)  could  not  be  practiced  on  the  simu¬ 
lator. 

Structured  Interviews.  Response?  to  the  13  questions  that  were  posed 
during  the  interviews  are  summarized  below. 

QUESTION  1:  What  were  the  instructional  features  of  the  simulator  that 
were  applicable  to  the  school's  training  course? 

Both  interviewees  stated  that  tne  3-D  module  was  applicable  and  that  it 
could  be  substituted  for  operational  equipment.  They  also  agreed  that  the 
malfunction  insertion  feature  was  useful  because  it  allowed  students  to 
troubleshoot  many  different  problems  and  because  it  eased  the  burden  of  the 
instructor.  Both  interviewees  noted  that  feedback  provided  to  the  student 
and  the  instructor  was  useful.  The  on-site  data  collector  felt  that  the 
sequence  of  simulator  lessons  was  applicable  since  it  was  flexible  and  could 
be  inserted  into  existing  self-paced  materials.  The  other  interviewee  liked 
the  flexibility  of  the  editing  system. 
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katings  made  by  the  interviewees  are  presented  below: 

RATINGS  MEAN  STANDARD 

(1  =  None,  7  =  Greatest)  N  RATING  DEVIATION 

Rate  the  +»"^ining  value 
of  these  ffc>.  ires  as  a 

whole.  2  5.5  .71 

Rate  the  potential 
training  value  of  these 

features  as  a  whole.  2  6.5  .71 

QUESTION  2:  What  were  the  instructional  features  of  the  simulator  that 
were  not  applicable  to  the  school's  training  course? 

Both  of  the  interviewees  thought  that  the  random  malfunction  selection 
feature  was  not  applicable  to  the  school's  training  course.  They  noted 
that  a  specific  subset  of  malfunctions  need  to  be  inserted  in  the  simulator 
when  training  certain  tasks,  but  this  degree  of  control  was  not  provided  by 
the  random  malfunction  feature.  The  interviewees  also  agreed  that  the  hard¬ 
copy  printout  of  student  performance  w's  inappropriate  since  the  school  does 
not  require  permanent  records  of  student  performance.  The  on-site  data 
collector  thought  that  the  slide  projector  unit  was  not  applicable  since  most 
of  the  diagrams  displayed  by  the  projector  were  also  available  on  student 
handouts.  This  interviewee  also  indicated  that  self-paced  lessons  were  not 
applicable  since  the  school  typically  uses  a  lockstep  method  of  instruction. 

QUESTION  3:  In  your  opinion,  which  features  of  the  lesson  presentation 
helped  make  the  simulator  lessons  interesting  to  the  students? 

Both  interviewees  agreed  that  feedback  provided  to  the  students  on  the 
CRT  helped  to  maintain  student  interest.  The  student  responder  unit  was  also 
seen  as  useful  for  maintaining  student  interest.  The  interviewees  concurred 
that  sel-p-paced  lessons  maintained  student  interest  since  the  students  were 
allowed  to  determine  how  much  time  to  devote  to  any  one  task.  The  on-site 
data  collector  stated  that  students'  hands-on  experience  with  the  3-D  module 
helped  make  the  lessons  interesting  to  students. 

Ratings  made  by  the  interviewees  are  presented  below: 

RATINGS  MEAN  STANDARD 

(1  =  None,  7  =  Greatest)  N  RATI  .  DEVIATION 

Rate  the  training  value 
of  these  features  as  a 

whole.  25  0 

Rate  the  potential 
'-r  ining  value  of  these 

features  as  a  whole.  2  5.5  .71 

Ql.'ESTION  4:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  particularly  effective? 


The  interviewees  agreed  that  the  monthly  and  wce^cly  check  procedures 
were  very  effective.  They  stated  that  students  trained  on  the  simulator 
performed  these  tasks  very  well  on  the  operational  equipment,  in  part, 
because  simulator  training  had  reduced  students'  fear  of  the  operational 
equipment.  The  on-site  data  collector  indicated  that  the  high  voltage 
lessons  presented  on  the  simulator  were  highly  effective  since  students 
normally  receive  no  training  for  these  tasks. 

Ratings  made  by  the  interviewees  are  presented  below: 

RATINGS  MEAN  STANDARD 

(1  =  None,  7  =  Greatest)  N  RATING  DEVIATION 

Rate  the  training  value  of 

these  lessons  as  a  whole.  2  6  1.41 

Rate  the  potential 

training  value  of  these 

lessons  as  a  whole.  27  0 

QUESTION  5;  Of  the  lessons  taught  by  the  simulator,  which  ones  did 
you  feel  were  ineffective? 

The  on-site  data  collector  felt  that  none  of  the  lessons  were  ineffective. 
The  training  specialist  felt  that  the  lessons  involving  the  noise  degeneration 
circuits  were  ineffective  because  low  fidelity  representations  of  components 
involved  in  these  circuits  were  included  in  the  simulator. 

QUESTION  6:  Were  there  any  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  being  presented  by  the  simulator  was  above  the  students? 

Both  interviewees  felt  that  the  lessons  involving  the  noise  degeneration 
circuits  were  too  difficult  for  the  students.  The  training  specialist  stated 
that  the  low  fidelity  representation  of  components  involved  in  these  circuits 
made  it  difficult  for  students  to  understand  these  lessons. 

QUESTION  7:  Were  there  aiiy  occasions  when  you  felt  that  the  difficulty 
level  of  the  material  being  presented  by  the  simulator  should  be  increased? 

The  on-site  data  collector  stated  that  none  of  the  material  presented 
by  the  simulator  was  too  easy  for  students.  The  training  specialist  felt 
that  the  high  voltage  lessons  were  too  simple  for  the  following  two  reasons: 

1)  the  lessons  did  not  simulate  malfunctions  for  all  high  voltage  components 
(cables  and  high  voltage  test  set  were  assumed  to  be  operating  correctly), 
and  2)  the  lessons  led  students  to  believe  that  a  specific  set  of  bad  readings 
could  only  be  caused  by  one  malfunction  when,  in  reality,  any  given  set  of 
bad  readings  could  be  caused  by  a  number  of  malfunctions. 

QUESTION  8:  What  hardware  features  of  the  simulator  made  it  more 
effective  than  conventional  training? 

The  interviewees  agreed  that  the  simulator  was  more  effective  than  con¬ 
ventional  training  because  a  variety  of  realistic  problems  can  be  easily 
inserted  into  the  simulator,  while  it  is  very  time  consuming  to  insert  the 
small  number  of  minor  problems  that  can  be  inserted  into  operational  equipment. 


The  interviewees  also  concurred  that  feedback  provided  to  students  made  the 
simulator  more  effective  than  conventional  training. 

The  on-site  data  collector  felt  that  the  physical  "'ut  of  the  3-D 
module  enhanced  performance  because  students  could  sta.  .^right  while 
troubleshooting  (students  must  maintain  awkward  positions  when  working  on 
operational  equipment).  This  interviewee  also  felt  that  the  low  voltages 
present  in  the  3-D  module  allowed  the  students  to  work  on  lessons  without 
fear  of  injuring  themselves  or  damaging  equipment  (potentially  lethal  voltages 
are  present  in  the  operational  equipment). 


Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

me;n 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

2 

6.5 

.71 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

2 

7 

0 

QUESTION  9:  What  were  the  features  that  made  this  simulator  easy  for 
the  instructors  to  operate? 

The  interviewees  agreed  that  the  instructor  panel  was  simple  to  operate, 
and  that  it  allowed  the  instructors  to  exercixe  great  flexibility  in  the  way 
that  lessons  were  presented  to  the  students.  Both  interviewees  felt  that  the 
reliability  of  the  device  was  good  (except  for  the  slide  projector  unit); 
instructors  did  not  spend  a  lot  of  time  repairing  the  simulator.  The  on¬ 
site  data  collector  stated  that  the  editing  system  was  easy  to  operate.  She 
note  that  this  system  allowed  instructors  to  make  various  changes  to  lessons 
which  expedited  student  progress. 


Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

(1  *  None,  7  *  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

2 

5.5 

.71 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

2 

6 

1.41 

QUESTION  10:  What  were  the 

features 

that  made 

this  simulator  easy  for 

the  students  to  operate? 

The  interviewees  agreed  that  the  student  responder  unit  facilitated 
operation  of  the  simulator. 


Ratings  for  this  feature  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 

whole. 

2 

4.5 

.71 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

2 

5 

1.41 

QUESTION  11:  What  aspects  of  the  device  would  be  appropriate  for  sub¬ 
stituting  for  actual  equipment? 

Interviewees  interpreted  the  question  as  "What  aspects  of  the  device 
made  it  appropriate  for  substituting  for  actual  equipment?".  Both  inter¬ 
viewees  felt  that  the  ease  of  inserting  realistic  malfunctions  made  the 
simulator  appropriate  for  substituting  for  operational  equipment.  The 
training  specialist  stated  that  the  reliability  of  the  device  and  its  high 
fidelity  also  made  it  substitutable  for  actual  equipment,  while  the  on-site 
data  collector  felt  that  the  self-paced  lessons  and  the  lack  of  dangerous 
voltages  made  the  simulator  appropriate  for  substituting  for  actual  equipment. 

QUESTION  12:  What  types  of  problems  did  students  have? 

Interviewees  agreed  that  students  experienced  problems  troubleshooting 
the  local  oscillator  tuning  motor,  the  cavity  tuning  motor,  and  the  arc 
detector  crystals  on  the  operational  equipment  because  these  components  were 
represented  on  the  simulator  with  low  physical  and  functional  fidelity.  The 
training  specialist  felt  that  students  experienced  problems  troubleshooting 
the  operational  equipment  because  a  great  deal  of  prompting  and  feedback  was 
present  throughout  training  on  the  simulator  but  no  feedback  was  present  when 
troubleshooting  the  operational  equipment. 

QUESTION  13:  How  would  you  employ  the  simulator  in  order  to  gain  maxi¬ 
mum  benefit  from  it? 

Both  interviewees  felt  that  the  simulator  should  be  used  in  a  school 
setting.  The  training  specialist  felt  that  training  on  the  simulator  should 
be  supplemented  by  training  on  operational  equipment.  The  on-site  data 
collector  stated  that  the  simulator  could  be  used  for  both  initial  training 
and  refresher  training. 

Ratings  made  by  the  interviewees  for  the  simulator  as  a  whole  are  pre¬ 
sented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  the  simulator  as  a  whole. 

2 

6 

1.41 

Rate  the  potential 
training  value  of  the 
simulator  as  a  whole. 

2 

7 

0 

Data  Collector  Observations.  A  variety  of  observations  about  the  simu- 
lator  were  made  by  the  data  collector  during  the  transfer-cf-training  experi¬ 
ment.  These  observations  are  summarized  below: 

0  The  slide  projector  unit  did  not  function  properly. 

Slides  frequently  appeared  at  the  wrong  time  or  did 
not  appear  at  all. 

0  On  several  occasions,  the  simulator  "locked-up," 
i.e.,  the  device  did  not  accept  inputs  from  the 
student  station,  instructor  station,  or  3-D  module. 

When  this  problem  occurred,  the  device  had  to  be 
turned  off  and  restarted. 

0  The  device  was  too  sensitive  to  movements  of  the 
3-D  module.  A  fair  amount  of  vibration  and  move¬ 
ment  is  usually  involved  in  removing  or  replacing 
a  component.  These  minor  movements  were  regarded 
as  errors  by  the  simulator  when,  in  fact,  they  are 
a  normal  part  of  remove/ replace  procedures. 

0  On  two  occasions,  the  simulator  ended  an  exercise 
before  the  student  had  finished  performing  required 
actions. 

0  Students  were  confused  when  correct  troubleshooting 
procedures  (verified  by  instructors)  were  entered 
on  the  student  responder  panel  and  they  received  a 
message  indicating  that  their  actions  were  incorrect. 

0  Students  always  received  feedback  on  the  student  CRT 
when  they  entered  correct  decisions  on  the  student 
responder  panel.  However,  feedback  was  not  always 
provided  when  students  entered  incorrect  decisions. 

Students  were  confused  because  they  did  not  know  if 
their  decision  was  correct  or  incorrect. 

0  Students  were  required  to  enter  a  troubleshooting 
procedure  on  the  student  responder  panel  before 
performing  the  procedure  on  the  3-D  module.  However, 
they  frequently  attempted  to  work  on  the  3-D  module 
without  entering  a  decision  on  the  student  responder 
panel.  Some  type  of  cue  for  students  to  enter 
decisions  on  the  responder  panel  may  be  appropriate. 

0  Instructors  were  able  to  use  the  editing  system 
effectively.  They  modified  messages  that  were 
presented  to  students  and  modified  the  lessons  so 
that  different  student  actions  were  accepted  as 
correct. 

0  The  "free  play"  mode  of  the  simulator  was  useful 
because  it  allowed  students  to  experiment  with 
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controls,  cables,  etc.  The  instructor  used  this 
feature  to  demonstrate  the  capability  of  the 
device  to  students. 

0  Instructors  were  able  to  start  (or  restart)  a 
training  session  at  any  point  in  a  lesson  by 
entering  a  few  commands  on  the  instructor 
station.  This  feature  was  valuable  to  instruc¬ 
tors  since  training  sessions  were  frequently 
interrupted.  If  this  feature  had  been  unavail¬ 
able,  students  (or  instructors)  would  have  been 
forced  to  repeat  the  entire  lesson  that  had  been 
interrupted. 

0  The  simulator  contained  a  "halt  on  student  errors" 
feature  that  stopped  students'  troubleshooting 
activities  after  they  had  committed  three  conse¬ 
cutive  errors,  and  a  "continue  on  student  errors" 
feature  that  allowed  students  to  troubleshoot 
regardless  of  the  number  of  errors  that  were 
committed.  These  features  can  be  activated  by 
instructors  very  easily.  Although  the  "continue 
on  student  errors"  feature  was  always  used  during 
the  transfer-of-training  study,  instructors  indicated 
that  the  "halt  on  student  errors"  feature  could  be 
very  useful . 

Discussion 


Comments  made  by  instructors,  students,  and  the  on-site  data  collector 
can  be  summarized  as  follows: 

Positive  Statements.  Instructors  and  students  hold  favorable  opinions 
of  the  device.  In  general ,  instructors  thought  that  the  device  was  easy  to 
learn  to  use.  They  also  indicated  that  the  device  did  a  good  job  of  training 
students  to  perform  troubleshooting  tasks.  Students  indicated  that  they 
enjoyed  working  with  the  device. 

"Hands-on"  troubleshooting  experience  is  a  highly  valued  device  feature. 
Respondents  repeatedly  stated  that  the  simulator  was  valuable  because  it 
allowed  students  to  gain  "hands-on"  experience  troubleshooting  high  voltage 
components.  Students  are  not  provided  the  opportunity  to  troubleshoot  the 
high  voltage  components  of  operational  equipment  because  lethal  voltages  are 
present  in  these  components  and  because  inserting  a  malfunction  into  high 
voltage  components  of  operational  equipment  seriously  damages  the  equipment. 
Respondents  also  noted  that  it  is  much  easier  to  insert  malfunctions  into  the 
simulator  than  it  is  to  insert  malfunctions  into  operational  equipment. 

Negative  Statement.  Low  fidelity  representations  of  certain  components 
of  the  3-D  module  reduced  the  effectiveness  of  the  simulator.  Respondents 
stated  that  they  disliked  the  low  fidelity  representation  of  the  tuning  motors, 
arc  detector  crystals,  and  certain  controls  and  displays.  They  felt  that 
these  low  fidelity  components  detracted  from  the  training  effectiveness  of 
the  simulator. 
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Experiment  6 

Three  students  from  the  24E  MOS  (Improved  Hawk  Fire  Control  Mechanic), 
four  students  from  the  24G  MOS  (Improved  Hawk  Information  and  Coordination 
Control  Mechanic),  and  three  students  from  the  24R  MOS  (Improved  Hawk  Master 
Mechanic)  were  trained  to  perform  troubleshooting  tasks  using  the  Grumman 
simulator.  These  students  were  then  tested  on  their  ability  to  perform 
troubleshooting  tasks  on  the  simulator  and  on  operational  equipment.  Data 
reported  in  this  document  for  Experiment  6  consist  of  responses  to  ques¬ 
tionnaires  and  structured  interviews  as  well  as  observations  made  by  the 
on-site  data  collector  during  the  course  of  the  experiment. 

Method 


Subjects.  Characteristics  of  subjects  who  provided  data  are  as 
follow: 

0  Initial  instructor  questionnaire.  Two  Non- 
commissioned  Officers  (NCOs)  who  were  instruc¬ 
tors  for  the  24  CIO  MOS  completed  the  ques¬ 
tionnaire. 

0  Course  developer  questionnaire.  One  civilian 
course  developer  for  the  24C10  MOS  completed 
this  questionnaire. 

0  Trainee  questionnaire.  Ten  students  from  three 
MOSs  who  had  received  training  on  the  Grumman 
simulator  completed  this  questionnaire  after 
they  had  been  tested  on  their  ability  to  perform 
troubleshooting  procedures  on  the  simulator  and 
on  the  operational  equipment. 

0  Structured  interviews.  Interviews  were  conducted 
with  a  course  developer,  three  civilian  training 
specialists,  and  the  on-site  data  collector. 

See  Table  7  for  a  listing  of  simulator  segments  that  were  evaluated. 

Materials.  The  questionnaires  and  structured  interview  forms  used  in 
this  experiment  were  identical  to  those  used  in  previous  experiments.  (See 
Appendix  A. ) 

Procedure.  Prior  to  filling  out  a  questionnaire  or  participating  in  an 
interview,  subjects  were  briefed  on  the  purpose  of  the  questionnaire  (or 
interview)  and  were  informed  of  the  types  of  questions  they  would  be  asked. 

Initial  instructor  questionnaires  were  completed  by  two  instructors 
after  they  completed  two  and  a  half  days  of  a  five-day  operator  training 
course.  This  course  could  not  be  completed  by  instructors  because  the 
device  malfunctioned  repeatedly. 
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Table  7 


Simulator  Lesson  Segments  Included  in  the 
Evaluation  of  the  Grumman  Device  at  Fort  Bliss 


Segment  Number 


Description 


1  Introduction  to  AMTESS 

2  Introduction  to  High  Voltage  Circuits  (Games) 

3  Introduction  to  Weekly  High  Voltage  Checks  for  High 

Voltage  Circuits 

4  Indicator  Relationships  (Meter  Responses  to  the  Condi¬ 

tions  of  Components) 

5  Last  Good/First  Bad  Method  of  Troubleshooting 

6  Introduction  to  High  Voltage  Power  Supply  Test  Set 

7  High  Voltage  Power  Supply  Test  Set  Procedures 

8  Problem  #1  -  Failed  Master  Oscillator  Filament  Power 

Supply  A4 

9  Problem  #2  -  Failed  Power  Amplifier  High  Voltage  Power 

Supply  A2 

10  Introduction  to  Troubleshooting 

11  Troubleshooting  the  High  Voltage  Regulator 

12  Troubleshooting  (no  feedback): 

Fault  11  -  Failed  Master  Oscillator  High  Voltage 
Power  Supply  A1 

Fault  10  -  Failed  High  Voltage  Regulator 
Fault  12  -  Failed  Power  Amplifier  Filament  Power 
Supply  A3 
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Weekly  Check  Procedures 


The  course  developer  was  asked  to  complete  a  questionnaire  for  each  of 
the  41  non-redundant  lesson  segments  presented  by  the  simulator.  (See  Table 
8;  although  the  table  lists  59  lesson  segments,  18  of  these  are  redundant.) 
However,  when  the  course  developer  was  available  to  review  simulator  segments, 
the  device  was  not  working  correctly,  and  when  the  device  was  working  cor¬ 
rectly,  the  course  developer  was  not  available  to  evaluate  the  device.  Thus, 
the  course  developer  completed  only  two  questionnaires  for  the  13  simulator 
segments  that  were  involved  in  the  transfer-of- training  study.  One  question¬ 
naire  was  completed  for  11  segments  that  addressed  high  voltage  procedures 
while  another  questionnaire  was  completed  for  the  segment  that  addressed 
weekly  check  procedures.  No  questionnaire  was  completed  for  the  segment 
that  involved  an  introduction  to  AMTESS. 

Ten  students  from  three  MOSs  completed  trainee  questionnaires  after  they 
completed  training  and  after  they  were  tested  on  their  ability  to  perform 
troubleshooting  tasks  on  the  simulator,  and  weekly  check  procedures  on 
operational  equipment.  Trainees  completed  questionnaires  independently. 

Five  individuals  completed  the  structured  interview  after  all  transfer- 
of-training  data  had  been  collected.  Interviews  were  conducted  on  an  indivi¬ 
dual  basis.  One  interview  was  self-administered.  For  the  remaining  four 
interviewees,  the  protocol  was  as  follows: 

0  The  interviewer  asked  the  interviewee  a  question. 

0  The  interviewer  prompted  the  interviewee  for  specific 
comments  about  the  simulator  if  the  interviewee  was 
initially  unresponsive. 

0  The  interviewer  asked  the  interviewee  to  rate  simulator 
features  (for  questions  1,  3,  4,  8,  9,  10,  11,  and  13). 

0  The  interviewer  reviewed  comments  with  the  interviewee 
to  ensure  correctness  of  recorded  responses. 

Certain  questions  ask  respondents  to  comment  on  specific  features  of 
the  Grumman  device.  These  features  are  described  in  Table  5. 

Results 


Initial  Instructor  Questionnaire.  Results  of  the  initial  instructor 
q ues tlonnaire  are  summarized  below. 

QUESTION  1:  How  easy  or  hard  is  it  to  learn  to  use  the  simulator? 
RATINGS:  1  =  very  hard,  5  =  very  easy 
N  =  2 


Response 

Number  of  Responses 

VERY  EASY 

1 

FAIRLY  EASY 

0 

AVERAGE 

1 

FAIRLY  HARD 

0 

VERY  HARD 

0 

TOTAL  2 
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Table  8 


Lesson  Segments  Presented  by  the  Grumman  Simulator 
in  the  Radar  Transmitter  Configuration 


Segment  Number 


Description 


0 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

ss 

56 

57 

58 


Introduction  to  AMTESS 

Introduction  to  High  Voltage  (HV)  Circuits 

Introduction  to  Weekly  Checks  for  High  Voltage  Circuits 

Indicator  Relationships  (Heter  Responses  to  the  Conditions  of  Conponents) 

Last  Good/First  Bad  Method  of  Troubleshooting 

Introduction  to  High  Voltage  Power  Supply  Test  Set  (HVPSTS) 

HVPSTS  Procedures 

Problem  #1:  Failed  Master  Oscillator  Filament  Power  Supply  A4 
Problem  #2;  Failed  Power  Amplifier  High  Voltage  Power  Supply  A2 
Introduction  to  Troubleshooting  Problem 
Problem  13:  Failed  High  Voltage  Regulator  (HVR) 

Problem  44:  Failed  Master  Oscillator  High  Voltage  Power  Supply 
(MOVHPS)  A1 

Problem  fS:  Failed  Power  Amplifier  Power  Supply  (PAPS)  A3 
Introduction  to  Troubleshooting  Problem 
Problem  43  Repeated  (Failed  HVR) 

Problem  44  Repeated  (Failed  MOHVPS  A1) 

Problem  45  Repeated  (Failed  PAPS  A3) 

Introduction  to  Troubleshooting  Problem 
Problem  43  Repeated  (Failed  HVR) 

Problem  44  Repeated  (Failed  MOHVPS  A1 ) 

Problem  45  Repeated  (Failed  PAPS  A3) 

Conclusion  to  High  Voltage  Circuits 

Introduction  to  Radio  Frequency  (RF)  Circuits 

RF  Clreults/Arc  Detection  Circuits  Relationships 

RF  Circuits  Problem  41 

RF  Circuits  Problem  42 

Introduction  to  Troubleshooting  Problem 

RF  Circuits  Problem  43 

RF  Circuits  Problem  44 

RF  Circuits  Problem  45 

Introduction  to  Troubleshooting  Problem 

RF  Circuits  Problem  43  (Reoeated) 

RF  Circuits  Problem  44  (Repeated) 

RF  Circuits  Problem  45  (Repeated) 

Introduction  to  Troubleshooting  Problem 
RF  Circuits  Problem  43  (Repeated) 

RF  Circuits  Problem  44  (Repeated) 

RF  Circuits  Problem  45  (Repeated) 

Conclusion  to  RF  Circuits 

Introduction  to  Arc  Detection  Circuits 

Arc  Detection  Circuits  Problem  41 

Arc  Detection  Circuits  Problem  42 

Weekly  Check  Procedure 

Noise  Degeneration  Circuits  Problem  41 

Noise  Degeneration  Circuits  Problem  42 

Introduction  to  Troubleshooting  Problem 

Noise  Degeneration  Circuits  Problem  43 

Noise  Degeneration  Circuits  Problem  44 

Noise  Degeneration  Circuits  Problem  45 

Introduction  to  Troubleshooting  Problem 

Noise  Degeneration  Circuits  Problem  43  (Repeated) 

Noise  Degeneration  Circuits  Problem  44  (Repeated) 

Noise  Degeneration  Circuits  Problem  45  (Repeated) 

Introduction  to  Troubleshooting  Problem 

Noise  Degeneration  Circuits  Problem  43  (Repeated) 

Noise  Degeneration  Circuits  Problem  (Repeated) 

Noise  Degeneration  Circuits  Problem  45  (Repeated) 

Credits 

Seventeen  (17)  Practice  Problems  Available 


*Slnce  evaluation  of  the  device  required  a  revision  of  the  sequence  In  which 
lesson  segments  were  presented,  lesson  segment  nunbers  do  not  match  those 
presented  In  Table  7. 
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The  mean  rating  for  all  instructors  was  4.0  (standard  deviation  =  1.41). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

One  instructor  stated  that  it  was  very  easy  to  learn  to  use  the  simulator 
because  the  briefing  provided  by  representatives  of  Seville/Burtek  was 
thorough  and  because  he  had  several  years  of  experience  with  the  Hawk  system. 
The  other  instructor  stated  that  the  simulator  malfunctioned  when  he  worked 
with  it.  He  indicated  that  the  computer  malfunctioned  when  he  attempted  to 
skip  certain  troubleshooting  procedures. 

QUESTION  3:  How  do  you  feel  about  the  simulator? 

RATINGS:  1  =  dislike  it  very  much,  5  *  like  it  very  much 

N  =  2 

Response  Number  of  Responses 

LIKE  IT  VERY  MUCH  0 

LIKE  IT  SOMEWHAT  2 

NO  STRONG  FEELINGS  ABOUT  IT  0 

DISLIKE  IT  SOMEWHAT  0 

DISLIKE  IT  VERY  MUCH  0 

TOTAL  I 

QUESTION  4:  Why  did  you  answer  Question  3  as  you  did? 

One  instructor  stated  the  simulator  added  a  new  dimension  to  learning 
and  to  teaching.  He  felt  that  the  device  saved  time,  energy,  and  manpower. 

The  other  instructor  felt  that  the  symptoms  for  certain  malfunctions  were 
inaccurate  and  that  this  could  be  misleading  to  students. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

One  instructor  felt  that  the  simulator  was  too  expensive.  (Since 
accurate  cost  projections  for  the  device  were  not  available,  this  comment 
was  apparently  based  on  rumor.)  He  also  felt  that  instructors  should  be 
trained  not  only  to  operate  the  simulator,  but  also  to  program  it,  repair  it, 
and  perform  maintenance  on  it.  The  other  instructor  felt  that  the  simulator 
needed  to  be  "de-bugged"  before  students  used  it. 

Course  Developer  Questionnaire.  One  course  developer  completed  two 
questionnaires.  One  questionnaire  addressed  11  segments  involving  high 
voltage  procedures  witile  the  other  questionnaire  addressed  one  segment 
involving  weekly  check  procedures. 

QUESTION  1:  Is  this  a  critical  task  which  needs  to  be  trained? 

RESPONSES:  Yes,  No 

N  =  1 
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The  course  developer  indicated  that  both  the  weekly  check  procedures  and 
the  high  voltage  tasks  were  critical  tasks  which  needed  to  be  trained.  The 
respondent  made  no  comments  about  his  decisions  concerning  task  criticality. 

QUESTION  2;  Is  this  task  currently  trained  on  operational  equipment? 

RESPONSES;  Yes,  No 

N  =  1 

Response  Number  of  Responses 

YES  50 

NO  50 

The  weekly  check  procedures  were  taught  at  the  School .  The  respondent 
stated  the  Air  Defense  School  was  unable  to  train  high  voltage  tasks  using 
operational  equipment. 

QUESTION  3:  How  difficult  is  it  to  perform  this  task? 

MTINGS:  1  =  very  easy,  5  =  very  difficult 

N  =  1 


Response 


Number  of  Responses 


VERY  DIFFICULT  0 
SOMEWHAT  DIFFICULT  0 
ABOUT  AVERAGE  IN  DIFFICULTY  1 
SOMEWHAT  EASY  1 
VERY  EASY  0 


TOTAL  2 


The  course  developer  indicated  that  the  weekly  check  procedures  were  some¬ 
what  easy  to  perform  and  that  the  high  voltage  procedures  were  about  average 
in  difficulty. 

QUESTION  4:  At  what  skill  level  should  a  trainee  perform  this  task  at 
the  end  of  training? 

Response  Number  of  Responses 

EXPERT  0 

LESS  THAN  EXPERT  2 

APPRENTICE  0 

BETTER  THAN  NOVICE  0 

NOVICE  0 

TOTAL  2 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

The  course  developer  felt  that  students  were  well -prepared  to  perform 
the  complex  task  of  troubleshooting  high  voltage  problems  on  operational 
equipment  after  they  completed  the  high  voltage  lessons  on  the  simulator. 
Providing  "hands-on"  training  for  hign  voltage  problems  was  one  of  the  most 
important  features  of  the  simulator.  It  may  not  be  cost-effective  or 
training-effective  to  use  the  simulator  to  train  a  fixed  procedure,  such  as 
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the  weekly  checks  procedure.  He  also  noted  that  the  simulator  malfunctioned 
often,  and  that  students  concluded  that  training  is  more  effective  on  opera¬ 
tional  equipment  than  it  is  on  the  Grumman  simulator. 

Trainee  Questionnaire.  Responses  of  the  10  students  who  were  trained 
with  the  simulator  are  summarized  below. 

QUESTION  1:  How  do  you  feel  about  the  simulator? 

RATINGS:  1  =  dislike  it  very  much,  5  =  like  it  very  much 

N  =  10 


Response  Number  of  Responses 

LIKE  IT  VERY  MUCH  5 

LIKE  IT  SOMEWHAT  4 

NO  STRONG  FEELINGS  ABOUT  IT  0 

DISLIKE  IT  SOMEWHAT  1 

DISLIKE  IT  VERY  MUCH  _0 

TOTAL  10 


The  mean  rating  for  all  students  was  4.3  (standard  deviation  =  .95). 

QUESTION  2:  Why  did  you  answer  Question  1  as  you  did? 

Several  students  liked  the  simulator  because  training  sessions  with  the 
device  were  more  interesting  than  conventional  training  sessions.  They  also 
liked  the  device  because  it  was  easier  to  understand  information  presented  by 
the  simulator  than  it  was  to  understand  information  presented  by  an  instruc¬ 
tor,  and  because  lessons  were  self-paced.  Many  students  disliked  the  fact 
that  the  simulator  malfunctioned  frequently.  Other  students  did  not  like 
self-paced  lessons. 

QUESTION  3:  What  specific  features  of  the  simulator  do  you  like  or  dis¬ 
like? 


The  lesson  that  was  presented  in  the  form  of  a  game  (identification  of 
high  voltage  components)  was  liked  by  most  students.  The  safety  of  the  3-D 
module,  lessons  involving  high  voltage  circuits,  the  touch  panel,  the  request’ 
help  feature,  the  block  diagrams  that  appeared  on  the  CRT,  the  3-D  module, 
the  lack  of  noise  generated  by  the  3-D  module,  and  the  cue  (chirp)  to  attend 
to  the  CRT  were  other  features  that  students  liked.  The  wide  variety  of 
problems  experienced  due  to  device  malfunctions  was  disliked  by  students. 

They  also  did  not  like:  the  request  help  feature,  specific  hardware  components 
such  as  the  high  voltage  cables,  the  touch  panel,  the  quality  of  photographs 
presented  in  Segment  1,  and  the  inability  to  complete  a  lesson  without  instruc¬ 
tor  intervention  after  they  have  made  several  consecutive  errors. 

QUESTION  4:  Why  do  you  like  or  dislike  these  features? 

Students  who  stated  that  they  liked  the  lesson  that  was  presented  as  a 
game  felt  that  this  lesson  was  interesting,  challenging,  and  effective. 
Respondents  indicated  that  they  liked  the  safety  of  the  3-D  module  (low 
voltages)  because  it  allowed  them  to  stop  worrying  about  injuring  themselves. 
One  student  stated  that  the  safety  of  the  3-D  module  created  a  relaxed 
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atmosphere  in  which  students  could  concentrate  on  their  work.  Students  liked 
lessons  involving  high  voltage  circuits  because  they  must  work  on  these  com¬ 
ponents  in  the  field.  Conventionally  trained  students  do  not  receive  "hands- 
on"  experience  with  high  voltage  pnblems  at  the  Air  Defense  School.  Tl.e 
touch  panel  feature  was  described  by  students  as  interesting  and  easy  to  use. 
Students  who  liked  the  request  help  feature  indicated  that  this  feature  was 
easy  to  use,  and  presented  useful  inforftiation.  Other  students,  however,  did 
not  like  this  feature.  One  student  stated  that  information  provided  by  this 
feature  did  not  always  provide  precise  information  (i.e.,.  location  of  com¬ 
ponents),  while  another  student  did  not  understand  how  to  use  this  feature. 

Many  students  disliked  simulator  features  because  these  features  mal¬ 
functioned  frequently.  Students  indicated  that  1)  correct  procedures  per¬ 
formed  on  the  3-D  module  were  recordea  as  errors  or  were  net  recorded  at  all, 

2)  the  audio  cue  to  attend  to  the  student  CRT  did  not  function  properly,  and 

3)  the  touch  panel  did  not  record  student  responses  reliably.  Other  students 
indicated  that  certain  controls  on  the  3-D  module  were  not  in  correct  loca¬ 
tions,  that  the  high  voltage  cables  were  difficult  to  attach,  that  the  audio 
cue  to  attend  to  the  student  CRT  was  annoying,  and  that  the  quality  of  photo¬ 
graphs  presented  in  Segment  1  t'as  poor.  One  student  stated  that  the  flow 
diagram  presented  in  Segment  1  wos  backwards. 

QUESTION  5:  Please  make  any  additional  comments  you  feel  are  appropriate. 

Several  students  stated  that  training  with  the  simulator  was  fast  and 
more  interesting  than  conventional  training.  Students  stated  that  the  simu¬ 
lator  could  be  used  for  refresher  training  as  well  as  for  advanced  individual 
training.  One  student  felt  that  the  device  would  be  useful  for  training 
radar  operators  as  well  as  mechanics.  Many  students  indicated  that  there 
are  many  problems  with  the  device  that  need  to  be  corrected.  One  student 
stated  that  his  concentration  was  broken  whenever  problems  developed  with  the 
device.  Another  student  felt  that  the  simulator  was  difficult  to  use  because 
device  malfunctions  required  him  to  repeat  certain  training  segments.  Several 
students  felt  that  although  "hands-on"  experience  with  high  voltage  trouble¬ 
shooting  procedures  was  beneficial,  some  of  these  procedures  were  simplified. 
They  felt  that  more  complex  high  voltage  procedures  are  necessary.  One  stu¬ 
dent  stated  that  it  was  easy  to  transfer  skills  from  the  simulator  to  opera¬ 
tional  equipment. 

Structured  Interview.  Responses  to  the  13  questions  posed  during  the 
structured  interview  are  summarized  below. 

QUESTION  1;  What  were  the  instructional  features  of  the  simulator  that 
were  applicable  to  the  school's  training  course? 

Three  of  the  four  interviewees  who  responded  to  the  question  stated  that 
the  3-D  module  was  applicable  to  the  school's  training  course.  They  felt 
that  the  device  could  be  substituted  for  operational  equipment.  Interviewees 
felt  that  performance  feedback  was  applicable.  The  on-site  data  collector 
noted  that  more  feedback  was  provided  at  the  beginning  of  the  lesson  while 
less  feedback  was  provided  at  the  end  of  a  lesson.  The  repeat  lesson  feature 
was  applicable  since  students  could  repeat  a  lesson  if  they  did  not  understand 
it.  Other  features  interviewees  found  to  be  applicable  included  the  self-paced 
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lessons,  the  universal  instructor,  the  instructor  CRT,  the  call  instructor 
feature,  automated  malfunction  insertion,  tlie  video  disc  system,  lesson 
arrangement,  and  the  hardcopy  printout  of  student  performance. 

Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS  MEAN  STANDARD 


(1  =  None,  7  =  Greatest) 

N 

RATINf 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 

whole. 

4 

4.5 

.58 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

4 

5.5 

1 

QUESTION  2:  What  were  the  instructional  features  of  the  simulator  that 
were  not  applicable  to  the  -chool's  training  rvirse? 

Two  interviewees  felt  that  the  "chirp"  used  as  a  cue  to  attend  to  the 
student  CRT  was  inapplicable  because  it  confused  students.  One  of  these 
interviewees  felt  that  the  "chirp"  was  annoying.  These  comments  contrast 
with  students'  responses  (page  71)  indicating  that  the  "chirp"  was  helpful. 

Several  interviewees  felt  that  the  automated  pre-lesson  check  was  not 
applicable.  One  interviewee  stated  that  it  was  an  awkward  way  of  establish¬ 
ing  communication  between  the  student  CRT  and  3-D  module,  while  another  felt 
that  this  feature  was  time  consuming.  Two  interviewees  felt  that  the  instruc¬ 
tor  CRT  was  cot  applicable.  One  interviewee  indicated  that  the  CRT  was  use¬ 
ful  only  when  entering  commands  for  enrolling  students,  repeating  a  lesson, 
or  skipping  lessons.  They  felt  that  the  information  presented  on  this  CRT 
was  not  useful.  These  interviewees  also  felt  that  the  "call  instructor" 
feature  was  not  applicable.  One  interviewee  stated  that  this  feature  slowed 
the  lesson  presentation  and  involved  the  instructor  unnecessarily.  Another 
interviewee  noted  that  students  could  not  "experiment"  with  meters,  switches, 
and  controls  because  the  call  instructor  message  would  appear  on  the  student 
CRT  and  the  lesson  would  stop  if  the  student  attempted  to  "experiment." 
Interviewees  felt  that  the  editing  system  was  not  applicable  because  it 
was  difficult  for  an  instructor  to  use  this  system.  They  felt  that  only 
expert  programmers  could  use  the  system.  Other  features  described  as 
inapplicable  include  self-paced  lessons,  the  video  disc  system,  and  the 
universal  instructor  feature. 

QUESTION  3:  Which  features  of  the  lesson  presentation  helped  make  the 
simulator  lessons  interesting  to  the  students? 

Most  interviewees  agreed  that  self -paced  lessons  were  interesting  to 
students.  The  interviewees  also  felt  that  the  request  help  feature  made  the 
lessons  interesting  to  students.  The  course  developer  stated  that  this 
feature  provided  students  with  infonnation  shortly  after  it  was  requested. 

One  interviewee  stated  that  students  W3re  interested  'n  lessons  in  which  they 
were  required  to  locate  components  on  the  3-D  module.  One  interviewee  stated 
that  performance  feedback  made  lessons  interesting  to  students  since  the 


variety  of  feedback  provided  by  the  device  allowed  students  to  know  when  they 
were  following  correct  procedures  and  when  they  were  commiting  errors.  Other 
features  mentioned  by  interviewees  include  the  games  in  Segment  1,  high  voltage 
test  set  procedures,  preprogrammed  malfunctions,  the  3-D  module,  and  the  video 
disc  system. 

Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

4 

4.75 

.96 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

4 

5.75 

.5 

QUESTION  4;  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  particularly  effective? 

Two  interviewees  indicated  that  Segment  2  (introduction  to  high  voltage; 
was  effective  because  students  were  taught  nomenclature,  parts  location, 
and  function.  They  also  stated  that  Segment  7  (High  Voltage  Power  Supply 
Test  Set  procedures)  was  effective  because  1)  test  procedures  were  presented 
in  a  step-by-step  manner,  2)  students  received  "hands-on"  experience  perform¬ 
ing  the  procedures,  and  3)  feedback  was  provided  throughout  the  segment.  Two 
interviewees  felt  that  Segment  8  (troubleshooting  the  Master  Oscillator  Fila¬ 
ment  Power  Supply  A4)  was  effective  because  the  student  was  able  to  use  the 
student  CRT  to  find  the  correct  symptom  and  perform  the  correct  troubleshoot¬ 
ing  tests.  Other  segments  that  interviewees  noted  as  effective  include 
Segment  5  (troubleshooting  last  good/first  bad).  Segment  6  (high  voltage 
power  supply  test  set).  Segment  8  (troubleshooting  the  PA  high  voltage  PS  A2), 
and  Segment  13  (weekly  check  procedures). 


Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 


whole. 

3 

6 

1 

Rate  the  potential 

training  value  of  these 

features  as  a  whole. 

3 

6.67 

.58 

QUESTION  5:  Of  the  lessons  taught  by  the  simulator,  which  ones  did  you 
feel  were  ineffective? 

Two  interviewees  indicated  that  Segment  4  (indicator  relationships)  was 
ineffective.  In  this  segment,  students  are  shown  the  symptoms  of  a  variety 
of  malfunctions,  but  are  not  required  to  correct  the  malfunctions.  The 
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interviewees  felt  that  this  segment  was  ineffective  because  students  were 
presented  with  a  great  deal  of  information  that  would  be  quickly  forgotten. 
Students  progressed  through  the  segment  very  quickly  because  they  did  not 
want  to  devote  a  great  deal  of  time  to  a  lesson  that  was  of  little  value. 

A  training  specialist  felt  that  Segment  2  (high  voltage  circuits  -  games) 
was  ineffective  because  that  segment  took  too  much  time  to  complete  and 
because  the  block  diagram  shown  on  the  student  CRT  in  this  segment  was 
different  from  the  block  diagram  provided  as  a  student  handout.  The  inter¬ 
viewee  felt  that  the  different  diagrams  confused  students.  Other  segments 
that  interviewees  found  to  be  ineffective  include  Segment  3  (weekly  high 
voltage  checks).  Segment  10  (introduction  to  troubleshooting),  and  Segment  6 
(high  voltage  power  supply  test  set). 

QUESTION  6:  Were  there  any  occasions  when  you  felt  the  difficulty  level 
of  the  material  presented  by  the  simulator  was  above  the  students? 

None  of  the  interviewees  felt  that  the  material  presented  by  the  simula¬ 
tor  was  too  difficult  for  students  to  understand. 

QUESTION  7:  Were  there  any  occasions  when  you  felt  the  difficulty  level 
of  the  material  being  presented  by  the  simulator  should  be  increased? 

Two  interviewees  felt  that  the  difficulty  level  of  material  presented 
by  the  simulator  did  not  need  to  be  increased.  One  of  these  interviewees 
stated  that  if  necessary,  however,  the  difficulty  level  of  lessons  could  be 
increased  by  removing  repetitive  information  from  the  lessons. 

One  interviewee  stated  that  the  difficulty  level  for  troubleshooting 
high  voltage  problems  needed  to  be  increased.  These  lessons  could  mislead 
students  into  believing  that  each  high  voltage  problem  is  always  caused  by 
a  single  failed  component  where  in  reality  these  problems  may  be  caused  by 
one  of  several  faulty  components  or  by  multiple  faulty  components.  The  inter¬ 
viewee  also  disliked  these  lessons  because  certain  components  (high  voltage 
cables,  wiring)  were  always  assumed  to  function  correctly  when  in  reality 
these  components  could  malfunction. 

QUESTION  8:  What  hardware  features  of  the  simulator  made  it  more 
effective  than  conventional  training? 

The  interviewees  described  a  variety  of  features  that  made  the  simulator 
more  effective  than  conventional  training.  These  features  include:  1)  rapid 
insertion  and  removal  of  malfunctions,  2)  durability  of  simulator  components, 
3)  standardized  symptoms  of  faulty  components  (symptoms  of  faulty  components 
may  vary  on  operational  equipment),  4)  reliability  of  symptoms  (symptoms  may 
disappear  on  operational  equipment  before  the  faulty  component  is  located), 

5)  safety  of  the  3-D  module  (high  voltages  are  not  present),  6)  accessibility 
of  components  of  the  3-D  module  (students  are  able  to  stand  when  working  on 
the  3-D  module  while  they  must  assume  an  awkward  position  when  working  on 
operational  equipment),  7)  the  hardcopy  printout  of  student  perfomance,  8) 
the  touch  panel,  and  9)  the  video  disc  system. 


Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

C'’'?-.None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

•••*  * 

Rate  the  training  value 
of  these  r?:;tures  as  a 


whole. 

4 

5.5 

.58 

!<ate  the  potential 

training  value  of  these 

features  as  a  whole. 

4 

7 

0 

QUESTION  9:  What  were  the  features  that  made  this  simulator  easy  for 
the  instructors  to  operate? 

Four  interviewees  felt  that  the  simulator  was  easy  to  start.  Students 
could  begin  using  the  device  after  the  operator  activated  a  few  switches  and 
buttons.  Three  interviewees  felt  that  the  device  was  easy  to  operate  because 
it  kept  track  of  the  lessons  that  students  completed.  These  interviewees 
noted  that  the  device  automatically  presented  the  student  with  the  appropriate 
lesson. 

Other  features  mentioned  by  interviewees  include  the  instructor  station, 
the  lesson  summary  that  appeared  at  the  beginning  of  each  lesson,  the 
reliability  of  the  system,  the  universal  instructor,  and  the  automated  pre¬ 
lesson  check. 

Ratings  made  by  the  interviewees  are  presented  below: 

MEAN  STANDARD 

N  RATING  DEVIATION 


5  5.8  .84 

Rate  the  potential 
training  value  of  these 

features  as  a  whole  5  6.4  .89 

QUESTION  10:  What  were  the  features  that  made  this  simulator  easy  for 
students  to  operate? 

All  of  the  interviewees  stated  that  the  touch  panel  made  the  simulator 
easy  for  students  to  operate.  Two  interviewees  felt  that  entering  responses 
on  the  touch  panel  was  less  threatening  to  students  than  entering  responses 
on  a  keyboard.  Four  interviewees  stated  that  the  audio  track  of  the  video 
disc  system  simplified  the  task  of  operating  the  simulator.  This  feature 
reduced  the  requirement  to  read  text  that  appeared  on  the  CRT.  Interviewees 
also  felt  that  the  audio  "chirp"  feature,  the  request  help  feature,  and  the 
3-D  module  made  the  device  easy  for  students  to  operate. 


RATINGS 

(1  =  None,  7  ~  Greatest) 

Rate  the  training  value 
of  these  features  as  a 
whole. 
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Ratings  made  by  the  interviewees 

are 

presented  below: 

RATINGS 

(1  =  None,  7  =  Greatest) 

N 

MEAN 

RATING 

STANDARD 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whole. 

5 

4.6 

.89 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

5 

5.8 

.84 

QUESTION  11:  What  aspects  of  the  device  would  be  appropriate  for  sub¬ 
stituting  for  actual  equipment? 

All  of  the  interviewees  who  responded  to  the  question  concluded  that 
the  3-D  module  can  be  substituted  for  actual  equipment.  Two  interviewees 
felt  that  the  safety  features  of  the  3-D  module  (low  voltage  and  minimal 
generation  of  heat)  make  it  ideal  for  substitution  for  actual  equipment. 

The  on-site  data  collector,  however,  stated  that  the  simulator  could  not 
completely  replace  actual  equipment.  She  felt  that  students  need  to  have 
some  training  experience  on  actual  equipment. 

Other  features  mentioned  by  interviewees  included  the  ease  of  inserting 
malfunctions  into  the  device  and  the  lessons  presented  by  the  device. 

Ratings  made  by  the  interviewees  are  presented  below: 


RATINGS 

MEAN 

STANDARD 

(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  these  features  as  a 
whol e . 

4 

5.25 

.96 

Rate  the  potential 
training  value  of  these 
features  as  a  whole. 

4 

6.5 

.58 

QUESTION  12:  What  types  of  problems  c’d  the  students  have? 

All  the  interviewees  agreed  that  the  reliability  of  the  device  created 
serious  problems  for  students.  The  device  would  frequently  "lock  up"  (i.e., 
would  not  accept  any  student  input)  which  required  students  to  repeat  the 
lessor  in  which  they  were  participating  when  the  lock  up  occurred.  Four 
interviewees  indicated  that  students  experienced  problems  with  the  touch  panel. 
Students  received  error  messages  even  though  they  touched  the  panel  at  the 
appropriate  location.  This  same  problem  occurred  with  the  3-D  module.  Stu¬ 
dents  received  error  messages  on  the  CRT  when  they  performed  correct  pro¬ 
cedures  oa  the  3-D  module. 

A  training  specialist  felt  that  students  were  required  to  spend  excessive 
amounts  of  time  adjusting  controls  on  the  3-D  module  because  the  device  was 


overly  sensitive.  Although  displays  on  the  3-D  module  indicated  that  a  control 
had  been  adjusted  correctly,  messages  on  the  student  CRT  indicated  that  further 
adjustment  was  necessary. 

Students  experienced  problems  because  the  call  instructor  feature  could 
not  be  disabled.  The  course  developer  stated  that  this  feature  created 
problems  because  it  could  halt  student  progress  even  though  very  minor  errors 
were  committed  by  the  student. 

Other  problems  cited  by  interviewees  include:  1)  the  poor  quality  of 
photographs  in  Segment  1;  2)  the  request  help  feature  did  not  always  provide 
useful  information;  3)  the  lessons  presented  on  the  simulator  involved  the 
use  of  an  outdated  TM  (weekly  check  procedure);  and  4)  the  system  frequently 
required  an  inordinate  amount  of  time  to  respond  to  student  input. 

QUESTION  13:  How  would  you  employ  the  simulator  in  order  to  gain  maxi¬ 
mum  benefit  from  it? 

All  of  the  interviewees  agreed  that  the  simulator  should  be  used  in  a 
classroom  setting.  The  course  developer  stated  that  the  simulator  should  be 
used  in  conjunction  with  actual  equipment.  Three  interviewees  stated  that 
under  ideal  conditions,  only  one  student  at  a  time  would  work  on  the  device, 
although  the  device  would  still  be  effective  if  two  students  used  it  at  the 
same  time. 

Ratings  made  by  the  interviewees  for  the  simulator  as  a  whole  are  pre¬ 
sented  below: 


RATINGS  MEAN  STANDARD 


(1  =  None,  7  =  Greatest) 

N 

RATING 

DEVIATION 

Rate  the  training  value 
of  the  simulator  as  a 

whole. 

3 

4.3 

1.53 

Rate  the  potential 
training  value  of  the 
simulator  as  a  whole. 

3 

6.67 

.58 

Data  Collector  Observations.  The  observations  made  by  the  on-site  data 
collector  during  the  evaluation  of  the  Grumman  device  are  summarized  below. 

0  The  training  device  frequently  refused  to  accept  input 
from  the  3-D  module,  the  student  touch  panel,  or  the 
instructor  keyboard.  When  such  a  "lock  up"  occurred, 
the  instructor  was  forced  to  turn  the  entire  system 
off  and  then  turn  the  system  on  again  (i.e.,  reboot 
the  system)  in  order  to  alleviate  the  "lock  up." 

0  When  an  error  message  appeared  on  the  student  CRT, 
students  were  required  to  follow  directions  in  the 
message  (for  example,  the  PA  circuit  breaker  is  off, 
turn  it  on)  before  they  could  continue  with  a  lesson. 

However,  instructors  (and  students)  were  not  always 
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able  to  clear  such  error  messages  from  the  CRT  in 
order  to  continue  a  lesson.  On  some  occasions, 
instructors  responded  to  the  error  message  appro¬ 
priately,  but  the  device  did  not  recognize  these 
corrective  actions.  On  other  occasions,  additional 
inappropriate  error  messages  appeared  on  the 
student  CRT  after  an  initial  error  message  had 
been  cleared.  On  still  other  occasions,  actions 
undertaken  by  an  instructor  in  order  to  clear  an 
initial  error  message  resulted  in  a  subsequent 
error  message  that  indicated  that  the  actions 
undertaken  by  the  instructor  were  incorrect. 

(For  example,  an  error  message  would  indicate 
that  switch  A  should  be  turned  off.  When  the 
switch  was  turned  off,  an  error  message  would 
appear  indicating  that  switch  A  should  be  turned 
on.)  Problems  such  as  those  described  above 
could  only  be  alleviated  by  rebooting  the  system. 

0  The  system  failed  to  present  an  entire  lesson  to 
certain  students,  that  is,  portions  of  segments 
were  skipped.  Since  the  device  was  not  designed 
to  allow  instructors  to  repeat  portions  of  segments, 
the  system  was  rebooted  when  this  error  occurred. 

0  When  the  system  was  rebooted  before  a  segment  was 
completed,  all  information  concerning  student  per¬ 
formance  in  the  current  segment  was  lost.  Further, 
after  the  system  was  rebooted,  students  were 
required  to  repeat  the  entire  segment  since  there 
was  no  mechanism  to  automatically  advance  to  the 
point  where  the  malfunction  occurred.  The  system 
was  rebooted  for  nine  of  the  10  students  involved 
in  the  evaluation.  The  mean  number  of  reboots  per 
student  was  4.1,  resulting  in  a  mean  of  over  two 
hours  of  lost  training  time  per  student. 

0  The  printout  of  student  performance  indicated  total 
elapsed  time  per  segment,  number  of  errors  committted 
per  segment,  and  a  performance  index  related  to 
elapsed  time  per  segment.  These  measures  of  per¬ 
formance  were  frequently  incorrect  or  misleading. 

The  time  measure  was  misleading  because  it  did  not 
account  for  system  reboots.  That  is,  no  record 
was  available  for  the  amount  of  time  spent  in  segments 
that  were  rebooted.  Further,  this  measure  was  fre¬ 
quently  inflated.  For  some  segments,  students  were 
unaware  that  they  were  required  to  perform  an  action 
(press  "next"  on  the  touch  screen)  in  order  to  end  a 
segment  and  thereby  terminate  timing. 

The  number  of  student  errors  recorded  by  the  training 
device  differed  substantially  from  the  number  of 
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errors  recorded  by  the  on-site  data  collector.  The 
device  displayed  error  messages  when  no  error  was 
comittted  and  it  failed  to  display  error  messages 
when  errors  were  committed.  Further,  many  error 
messages  appeared  when  instructors  attempted  to 
rectify  legitimate  student  errors  or  device  mal¬ 
functions.  These  errors,  committed  by  instructors, 
were  included  as  part  of  the  student  performance 
record.  The  device  also  failed  to  account  for 
errors  committed  during  the  first  time  through 
segments  that  were  repeated  because  system  reboots 
were  required. 

0  Students  experienced  a  variety  of  problems  operating 
controls  on  the  3-D  module.  The  standby,  radiate, 
and  off  pushbuttons  had  to  be  pressed  for  an  extended 
period  of  time  before  the  device  recognized  that  these 
buttons  had  been  activated.  Similarly,  the  device  did 
not  recognize  student  adjustment  of  the  PA  beam  control, 

MO  beam  control,  PA  filament  control,  or  the  MO  fila¬ 
ment  control  unless  students  made  rapid,  gross  adjust¬ 
ments  . 

0  The  touch  panel  was  unreliable.  In  some  segments 
students  were  required  to  touch  certain  locations  on 
the  CRT  in  order  to  progress  through  a  lesson.  In 
some  instances,  students  appeared  to  press  the  correct 
location  on  the  CRT,  but  were  provided  feedback  indicat¬ 
ing  that  their  response  was  incorrect.  This  type  of 
error  occurred  most  frequently  when  the  angle  between 
the  student's  finger  and  the  touch  panel  was  signifi¬ 
cantly  greater  or  less  than  90  degrees. 

0  In  some  instances,  mate^'-ial  presented  to  students  on 
the  CRT  was  degraded  or  incorrect.  Degraded  displays 
included  poor  quality  video  frames,  and  computer¬ 
generated  text  that  appeared  near  the  edges  of  the  CRT 
{the  bevel  surrounding  the  CRT  obstructed  text  that 
appeared  near  the  edges  of  the  CRT).  Students  were 
presented  with  inaccurate  diagrams  and  text  on  the  CRT. 

Discussion 

Opinions  concerning  the  Grumman  device  at  Fort  Bliss  are  sunmarized  in 
a  series  of  positive  and  negative  statements  that  appear  below. 

Positive  Statements.  Instructors,  course  developers,  and  trainees  hold 
favorable  opinions  of  the  Grumman  simulator!  Instructors  indicated  that  they 
had  little  difficulty  learning  to  operate  the  simulator.  The  course  developer 
indicated  that  the  simulator  curriculum  addressed  critical  tasks  and  that 
students  trained  with  the  simulator  would  perform  well  on  operational  equip¬ 
ment.  Trainees  indicated  that  they  enjoyed  working  with  the  simulator  and 


felt  that  they  were  well -prepared  to  work  on  operational  equipment.  All 
three  of  these  groups  of  respondents,  however,  felt  that  the  utility  of  the 
device  was  limited  by  its  low  reliability. 

The  simulator  is  safer  than  operational  equipment.  Several  interviewees 
and  the  course  developer  indicated  the  students  were  less  likely  to  injure 
themselves  on  the  simulator  than  on  operational  equipment  because  low  voltages 
are  present  in  the  simulator.  Students  felt  that  the  simulator  was  less 
threatening  than  operational  equipment  because  lethal  voltages  are  not  present. 

Certain  automated  features  are  valuable.  Responses  to  several  different 
questions  indicated  that  various  automated  device  features  were  regarded  as 
valuable.  These  features  include  the  automated  pre-lesson  check,  request 
help,  feedback,  and  automatic  malfunction  insertion. 

The  simulator  allows  students  to  practice  tasks  that  cannot  be  practiced 
on  operational  equipment.  Students,  instructors,  and  course  d e velopers 
praised  the  device  for  its  capability  to  train  students  to  troubleshoot  high 
voltage  components  of  the  transmitter.  They  indicated  that  conventionally 
trained  students  are  not  taught  this  important  skill  because  insertion  of 
malfunctions  into  operational  equipment  is  time  consuming,  presents  a  safety 
hazard,  and  damages  equipment. 

Negative  Statements.  The  device  frequently  malfunctioned.  The  predomi¬ 
nant  criticism  of  the  device  was  its  low  reliability.  Various  kinds  of  hard¬ 
ware  and  software  failures  were  identified  by  trainees,  the  course  developer, 
and  instructors.  These  failures  caused  numerous  delays  in  training  sessions. 

Rebooting  is  a  poor  method  for  restarting  a  lesson.  In  many  cases, 
rebooting  was  the  on^  method  avai Tab! e  to  correct  a  system  malfunction. 

When  the  system  is  rebooted,  all  information  pertaining  to  the  current  lesson 
segment  is  lost.  Students  were  frustrated  because  frequent  rebooting  required 
them  to  repeat  lesson  segments. 

The  instructor  CRT  provided  little  valuable  information.  Instructors 
had  little  use  for  the  information  presented  on  the  instructor  CRT.  They  did 
not  understand  the  information  that  was  presented  on  the  CRT  and  the  informa¬ 
tion  they  desired  (student  performance)  was  not  presented  on  this  CRT. 

Lessons  are  inflexible.  Instructors  were  unable  to  change  the  sequence 
in  which  lessons  appeared.  Students  did  not  like  the  fact  that  they  could 
not  repeat  segments  (or  parts  of  segments)  or  omit  segments  (or  parts  of 
segments)  on  request. 


DISCUSSION  AND  CONCLUSIONS 

One  of  the  objectives  of  the  AMTESS  program  is  to  develop  a  family  of 
training  devices  that  is  applicable  to  a  range  of  training  applications.  For 
example,  AMTESS  devices  should  be  capable  of  training  both  mechanical  and 
electronic  maintenance  tasks.  By  comparing  the  comments  of  respondents  across 
experiments  for  each  device,  we  can  determine  if  the  perceived  effectiveness 
of  the  device  (and  specific  device  features)  varied  according  to  training 
application  (automotive  or  radar). 


The  objective  of  the  following  discussion  is  to  determine  (for  each 
device)  the  extent  to  which  the  opinions  of  students,  instructors,  and  course 
developers  varied  across  the  radar  and  automotive  applications. 

Table  9  displays  the  positive  and  negative  statements  that  were  presented 
at  the  end  of  each  of  the  six  experiments.  A  comparison  of  the  statements 
made  at  the  end  of  Experiments  1  and  2  (the  automotive  MOSs)  reveals  many 
similarities.  In  both  cases,  the  simulator  was  generally  well -received  by 
students,  instructors,  and  course  developers.  Further,  several  of  the 
features  that  were  rated  highly  in  the  63B30  context  (malfunction  insertion, 
performance  monitoring)  were  also  rated  highly  in  the  63W10  context. 

The  63B30  respondents  seemed  to  emphasize  the  safety  value  of  the  device 
more  than  the  63W10  respondents.  This  may  be  a  function  of  the  fact  that 
tasks  performed  by  63B30  students  during  the  device  evaluation  (adjust  alter¬ 
nator  belt,  remove/replace  starter  motor,  troubleshoot  an  oil  pump  failure, 
and  inspect  the  electrical  system)  were  more  difficult  and  dangerous  than 
those  performed  by  63W10  students  during  the  device  evaluation  (troubleshoot, 
remove  and  replace  an  oil  pump). 

The  63B30  and  63W10  respondents  also  made  similar  negative  comments 
about  the  Seville/Burtek  device.  In  both  cases,  respondents  disliked  the 
fact  that  set  up  and  check  out  of  the  STE/ICE  equipment  was  not  included  in 
the  simulator  curriculum.  The  reliability  of  the  device  was  criticized  in 
both  cases,  as  was  the  sensitivity  of  the  3-D  module. 

The  63B30  respondents  criticized  the  device  as  being  difficult  for 
students  to  operate  because  there  were  many  stimuli  demanding  attention. 
Although  the  63W10  instructor  stated  that  some  students  experienced  difficulty 
operating  the  device,  he  attributed  this  difficulty  to  the  abbreviated 
introduction  to  the  device  that  was  provided  to  students.  In  general,  the 
63W10  students  may  have  experienced  fewer  problems  operating  the  device 
than  the  63B30  students  because  they  performed  one  relatively  simple  task 
while  63B30  students  performed  several  complicated  tasks. 

More  negative  comments  about  the  low  physical  fidelity  of  the  3-D  module 
were  elicited  from  the  63B30  respondents  than  from  the  63W10  respondents. 

The  negative  comment  from  63B30  respondents  is  a  function  of  the  low  physical 
fidelity  of  components  associated  with  the  starter  motor  remove/replace  task. 
Since  63W10  respondents  did  not  perform  this  task,  they  did  not  make  similar 
comments . 

The  positive  comments  about  the  Seville/Burtek  device  that  were  made  in 
the  context  of  the  63W10  and  63B30  MOSs  (the  automotive  MOSs)  were  similar 
to  each  other,  but  somewhat  different  from  the  statements  presented  at  the 
end  of  Experiment  5,  which  involves  electronic  troubleshooting.  The  out¬ 
standing  aspect  of  the  Seville/Burtek  device  noted  during  Experiment  5  was 
its  ability  to  provide  "hands-on"  training  for  high  voltage  problems  which 
cannot  be  taught  using  operational  equipment.  Students  and  instructors 
repeatedly  indicated  that  this  capability  was  quite  valuable.  Since  students 
know  that  they  will  be  required  to  troubleshoot  high  voltage  components  of 
the  Hawk  radar  in  the  field  (a  very  dangerous  task),  they  appreciate  the 
opportunity  to  practice  this  task  in  a  non- threatening  environment.  Although 
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Table  9 


Summary 

Positive: 

Negative: 

Positive: 

Negative: 

Positive: 

Negative: 

Positive: 

Negative: 

Positive: 

Negative: 

Positive: 

Negative: 


of  Positive  and  Negative  Statements  about  the  AMTESS  Devices 


—  Sevi lle/Burtek  Device  Experiment  1  (MOS  63B3Q)  — 

Respondents  hold  favorable  opinions  of  the  simulator. 

Ease  of  inserting  malfunctions  is  valuable. 

Performance  monitoring  is  valuable. 

The  simulator  is  safer  than  operational  equipment. 

Students  were  confused  by  the  materials  to  which  they  must  attend. 
Physical  fidelity  of  the  3-D  module  is  too  low  for  certain  remove/ 
replace  tasks. 

The  reliability  and  durability  of  the  device  should  be  increased. 
Lessons  did  not  include  STE/ICE  set-up  and  check-out. 

The  3-D  module  is  too  sensitive. 

—  Sevi lle/Burtek  Device  Experiment  2  (MOS  63W10)  — 

Respondents  hold  favorable  opinions  of  the  simulator. 

Respondents  liked  features  including  feedback,  procedural i zed 

instructions,  slide  projector  unit,  malfunction  insertion. 

Lessons  did  not  include  STE/ICE  set-up  and  check-out. 

The  3-D  module  is  too  sensitive. 

The  durability  of  the  device  is  low. 

—  -  Sevi lle/Burtek  Device  Experiment  5  (MOS  24C10)  — 

Respondents  hold  favorable  opinions  of  the  device. 

"Hands-on"  troubleshooting  is  a  highly  valued  device  feature. 

Low  fidelity  components  of  the  3-D  module  reduced  device  effective 
ness. 

Grumman  Device  Experiiront  3  (MOS  63H30)  — - 
The  ability  to  perform  troubleshooting  tasks  on  the  3-D  module  is 
a  valuable  feature. 

The  video  disc  system  is  an  effective  motivating  feature. 

The  device  frequently  malfunctions. 

Lessons  arc  inflexible. 

Some  lessons  are  too  simple  or  ’nappropriate. 

The  student  performance  record  i:  of  little  value. 

System  response  time  is  too  slow. 

— Grumman  Device  Experiment  4  (MOS  63D30)  — 

Students  liked  features  including  the  3-D  module,  the  request  help 
feature,  the  video  disc  system,  procedural i zed  lessons,  and 
lessons  addressing  STE/ICE. 

The  device  frequently  malfunctioned  and  operated  too  slowly. 

-—  Grumman  Device  Experiment  6  (MOSs  24E,  G,  R)  --- 
Respondents  hold  favorable  opinions  of  the  device. 

The  device  is  safer  than  operational  equipment. 

Automated  features  (request  help,  pre-lesson  check,  feedback, 
malfunction  insertion)  are  valuable. 

The  device  allows  students  to  practice  tasks  that  cannot  be 
practiced  on  operational  equipment. 

The  device  frequently  malfunctioned. 

Rebooting  is  a  poor  method  for  restarting  a  lesson. 

The  instructor  CRT  provides  little  valuable  information. 

Lessons  are  inflexible. 


the  Seville/Burtek  device  also  allows  63W10  and  63B30  students  an  opportunity 
to  practice  tasks  that  are  not  included  in  conventional  training  (due  to 
potential  equipment  damage),  this  feature  of  the  device  is  less  important 
for  these  automotive  MOSs  because  the  tasks  are  not  dangerous  to  the  student 
and  the  tasks  comprise  a  small  part  of  the  automotive  curriculum  whereas  the 
high  voltage  tasks  comprise  a  substantial  aspect  of  the  radar  curriculum. 

The  negative  comments  for  the  63W10  and  the  63B30  respondents  are  also 
similar  to  each  other  and  disparate  from  the  negative  comment  from  the 
24C10  respondents.  Respondents  from  both  automotive  MOSs  commented  negatively 
on  the  lack  of  STE/ICE  lessons,  the  high  sensitivity  of  the  3-D  module  to 
remove/ replace  activities  and  the  low  reliability  of  the  device.  The  first 
two  comments  do  not  apply  to  the  Seville/Burtek  device  at  Fort  Bliss  since 
it  does  not  involve  STE/ICE  or  remove/replace  activities.  Although  the 
slide  projector  unit  fc;iled  at  Fort  Bliss,  these  failures  were  apparently 
less  severe  than  those  at  APG.  Further,  the  low  reliability  of  the  device 
at  APG  also  addresses  failure  of  components  (bolts,  coils)  involved  in 
remove/ replace  activities.  These  failures  did  not  occur  at  Fort  Bliss  since 
remove/replace  tasks  were  not  performed. 

The  63B30  respondents  at  APG  and  the  respondents  at  Fort  Bliss  agreed 
that  the  low  physical  fidelity  of  certain  components  of  the  3-D  modules 
detracted  from  the  effectiveness  of  the  device.  This  comment  applies  to 
quite  different  tasks  at  the  two  locations.  At  APG,  the  lack  of  a  propeller 
shaft  and  frame  on  the  3-D  module  made  the  starter  motor  remove/ replace  task 
simple  to  perform  on  the  simulator.  Students  were  not  well-prepared  to  per¬ 
form  this  task  on  operational  equipment,  however,  when  they  were  required  to 
maneuver  around  obstacles  (propeller  shaft  and  frame)  not  represented  on  the 
simulator.  At  Fort  Bliss,  students  indicated  that  they  experienced  difficulty 
troubleshooting  the  local  oscillator  tuning  motor,  cavity  tuning  motor,  and 
arc  detector  crystals  because  these  components  were  represented  on  the  simu¬ 
lator  with  low  physical  and  functional  fidelity. 

The  similarities  and  differences  between  the  comments  made  by  respondents 
at  APG  and  Fort  Bliss  for  the  Grumman  device  parallel  those  for  the  Seville/ 
Burtek  device  in  that  comments  about  the  automotive  MOSs  are  somewhat  similar 
to  each  other  and  different  from  comments  about  the  missile  MOSs. 

Positive  comments  about  the  Grumman  device  in  Experiment  3  centered 
around  two  valuable  device  features:  troubleshooting  on  the  3-D  module, 
and  the  video  disc  system.  Comments  derived  from  Experiment  4  are  limited, 
but  they  also  address  device  features  including  the  3-D  module  and  the  video 
disc  system. 

The  overriding  negative  comment  for  both  Experiment  3  and  Experiment  4 
concerns  the  reliability  of  the  device.  In  both  experiments,  students  and 
instructors  stated  that  device  malfunctions  resulted  in  numerous  delays. 

Other  negative  comments  reported  in  Experiment  3  (inflexible  lessons,  simple/ 
inappropriate  lessons,  insufficiently  detailed  student  performance  record) 
were  not  reported  in  Experiment  4.  This  difference  is  artifactual  since  much 
more  data  was  reported  in  the  former  experiment  than  in  the  latter. 
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As  was  the  case  with  the  Seville/Burtek  device  at  Fort  Bliss,  the  most 
important  feature  of  the  Grumman  device  at  Fort  Bliss  is  its  ability  to  pro¬ 
vide  safe,  hands-on  troubleshooting  practice  for  high  voltage  problems.  These 
problems  cannot  normally  be  taught  with  operational  equipment. 

The  list  of  automated  features  that  were  rated  highly  by  the  respondent 
at  Fort  Bliss  overlaps  considerably  with  the  features  that  were  rated  highly 
in  the  automotive  MOSs.  However,  the  video  disc  system,  a  major  component 
of  the  Grumman  device,  was  rated  more  highly  by  respondents  in  Experiments  3 
and  4  than  by  respondents  in  Experiment  6.  Respondents  in  Experiments  3  and 
4  felt  that  the  high  resolution  graphics  and  audio  track  of  the  video  disc 
system  were  helpful  to  soldiers  because  they  possessed  minimal  reading  skills. 
The  video  disc  system  may  not  have  been  valued  as  highly  at  Fort  Bliss 
because  the  soldiers  participating  in  the  study  possessed  considerable  skill 
in  reading  and  interpreting  schematics,  flow  charts,  etc.  Thus,  the  video 
disc  system  may  have  been  "overkill”  at  Fort  Bliss. 

The  negative  comments  about  the  Grumman  device  at  Fort  Bliss  are  similar 
to  the  negative  comments  about  the  device  at  APG.  Frequent  device  malfunc¬ 
tions  and  the  subsequent  requirement  to  reboot  the  system  are  the  predominant 
negative  comments.  Respondents  at  Fort  Bliss  and  at  APG  both  noted  the 
inflexible  nature  of  lesson  segments.  Additionally,  respondents  at  Fort  Bliss 
noted  the  limited  value  of  the  instructor  CRT.  This  weakness  may  have  been 
noticed  to  a  greater  degree  at  Fort  Bliss  than  at  APG  because  there  was  great 
interest  in  modifying  the  lesson  arrangement  at  Fort  Bliss,  but  not  at  APG. 

In  summary,  respondents  in  -the  radar  and  automotive  MOSs  generally  made 
the  same  type  of  comments  about  the  devices.  There  are,  however,  several 
notable  exceptions: 

1.  For  both  devices,  respondents  at  Fort  Bliss  emphasized 
the  simulators'  capability  to  provide  hands-on  practice 
to  a  greater  extent  than  did  respondents  at  APG. 

2.  For  the  Grumman  device,  respondents  at  APG  commented  on 
the  value  of  the  video  disc  system  to  a  greater  extent 
than  di  '  respondents  at  APG. 

3.  The  63B30  students  at  APG  (Seville/Burtek  device) 
reported  confusion  with  respect  to  the  number  of 
stimuli  to  which  they  were  required  to  attend. 

This  negative  comment  was  not  made  by  63W10 
students  or  by  students  at  Fort  Bliss. 

The  preceding  discussion  compared  comment.^  made  by  respondents  in 
different  MOSs.  While  this  information  is  valuable,  it  does  not  provide  an 
assessment  of  the  conceptual  approaches  undertaken  by  the  two  device  manu¬ 
facturers.  The  purpose  of  this  section  of  the  report  is  to  integrate  the 
results  of  all  six  experiments  in  order  to  determine  the  benefits  and  liabili¬ 
ties  of  the  conceptual  approach  utilized  by  Grumman  and  Seville/Burtek. 


Benefits  of  the  Gruircnan  Approach 


The  Grumman  approach  can  be  characterized  as  one  that  incorporates  recent 
advances  in  microelectronics  and  video  storage.  Respondents  at  APG  indicated 
that  they  were  favorably  impressed  by  the  video  disc  approach  used  in  the 
Grumman  device.  The  still  and  motion  video  frames  presented  on  the  student 
CRT  were  useful  at  both  locations.  The  sound  track  of  the  video  disc  system 
was  especially  useful  at  APG  since  the  soldiers  who  used  the  device  at  APG 
possessed  poor  reading  skills.  The  touch  panel  simplified  the  students' 
tasks  and  added  to  their  enthusiasm  for  using  the  training  device.  The 
component  location  "games"  included  as  part  of  the  curriculum  at  Fort  Bliss 
also  seemed  to  heighten  student  interest  in  the  simulator. 

A  second,  and  perhaps  more  important,  benefit  of  the  Grumman  approach, 
is  the  ability  of  the  device  to  train  students  to  perform  tasks  that  cannot 
be  practiced  on  operational  equipment.  This  benefit  was  especially  important 
at  Fort  Bliss  since  it  is  highly  useful  for  radar  mechanics  to  practice 
troubleshooting  high  voltage  problems. 


Liabilities  of  the  Grumman  Approach 

The  predominant  liability  of  the  Grumman  approach  is  the  very  low 
reliability  of  the  training  device.  This  low  reliability  prevented  instruc¬ 
tors  from  understanding  how  to  operate  the  device  (the  effectiveness  of 
operator  training  sessions  was  seriously  limited  by  device  malfunctions), 
caused  numerous  delays  in  training  sessions,  and  adversely  affected  student 
and  instructor  attitude  towards  working  with  the  device.  Although  it  is 
acknowledged  that  the  reliability  of  a  breadboard  device  cannot  be  expected 
to  be  as  great  as  the  reliability  of  a  production  model,  the  device  must 
function  well  enough  to  demonstrate  its  capabilities.  If  the  reliability  of 
the  device  cannot  be  improved  significantly  without  substantial  effort,  then 
the  utility  of  the  device  is  questionable. 

Opinions  solicited  from  individuals  at  both  locations  indicate  that  the 
device  is  inflexible.  This  inflexibility  is  manifested  in  several  ways: 

1)  it  would  be  costly  and  time  consuming  to  change  the  material  that  is  pre¬ 
sented  by  the  video  disc  system,  2)  student  progress  through  each  segment 
more  closely  resembles  lockstep  training  than  it  does  self-paced  training, 

3)  the  order  in  which  students  participate  in  training  segments  cannot  be 
readily  changed,  and  4)  the  call  instructor  feature  cannot  be  disabled  by 
an  instructor  while  a  segment  is  in  progress. 

The  manner  in  which  malfunctions  are  inserted  and  removed  from  the  device 
is  awkward  and  time  consuming.  For  example,  at  the  start  of  the  weekly  check 
procedures,  an  instructor  is  required  to  spend  a  considerable  amount  of  time 
ensuring  that  controls  on  the  3-D  module  are  set  to  correct  positions.  As 
a  student  progresses  through  a  lesson,  these  controls  are  set  to  a  variety  of 
new  positions  requiring  the  instructor  to  correctly  reset  the  controls  again 
after  the  lesson  has  been  completed.  This  procedure  of  setting  and  resetting 
controls  wastes  a  great  deal  of  instructors'  time. 
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As  was  the  case  with  the  Grumman  device,  the  Seville/Burtek  device  allows 
students  to  practice  tasks  that  they  could  not  practice  on  operational  equip¬ 
ment.  Although  this  feature  is  especially  important  for  training  students 
to  perform  high  voltage  tasks  on  the  radar  transmitter  at  Fort  Bliss,  it  was 
also  found  to  be  appropriate  at  APG. 

Data  gathered  from  both  locations  on  both  devices  indicated  that  the 
device  was  quite  flexible.  Instructional  materials  presented  to  students 
by  the  slide  projector  or  the  student  CRT  can  be  updated  easily.  Students 
may  skip  steps  in  a  lesson  (if  the  instructor  chooses)  and  may  complete 
lessons  in  any  sequence  desired  by  the  instructor.  Further,  the  halt  on 
student  error  feature  can  be  disabled  by  an  instructor  during  the  course 
of  a  lesson. 

The  Seville/Burtek  device  was  found  to  be  fairly  reliable.  Although 
difficulties  with  the  slide  projector  unit  were  experienced,  these 
difficulties  did  not  seriously  delay  training  sessions. 

Insertion  of  malfunctions  is  simple  and  efficient.  At  the  beginning  of 
the  weekly  check  procedure,  for  example,  instructors  spend  a  minimal  amount 
of  time  setting  controls  on  the  3-D  module  out  of  tolerance.  As  students 
progress  through  a  lesson,  they  must  set  these  controls  back  to  specific 
correct  settings.  Thus,  it  is  the  student  rather  than  the  instructor  who 
must  expend  effort  setting  controls  to  their  correct  positions. 


Liabilities  of  the  Seville/Burtek  Approach 

At  APG  (especially  the  63B30  MOS),  it  seemed  that  there  were  too  many 
stimuli  that  required  student  attention.  Further,  procedures  for  entering 
decisions  on  the  student  station  were  somewhat  complicated.  A  different 
approach  for  presenting  information  to  students  and  accepting  information 
from  students  appears  warranted.  That  is,  the  number  of  stimuli  to  which 
students  must  attend  should  be  decreased. 

The  durability  of  the  Seville/Burtek  device  should  be  increased. 

If  the  device  is  to  incorporate  an  effective  remove/replace  capability, 
the  helical  coils,  bolts,  etc.  must  be  "hardened"  to  withstand  the  rigors 
of  normal  use. 

Conclusions 


The  data  reported  in  this  volume  of  the  report  are  diverse  and  substan¬ 
tial.  Individuals  with  very  different  backgrounds  and  attitudes  towards 
training  devices  provided  information  about  the  AfITESS  devices.  Two  conclu¬ 
sions  can  be  drawn  from  these  diverse  data; 

0  The  conceptual  approaches  to  generic,  modular  maintenance 
training  devices  developed  by  Grumman  and  Seville/Burtek 
are  viable.  Although  deficiencies  in  both  devices  were 
noted,  data  presented  in  this  report  indicate  enthusiasm 
for  this  type  of  training  device. 
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The  utility  of  the  devices  (and  specific  device 
features)  varies  according  to  training  application 
Additional  research  is  required  to  identify  the 
conditions  under  which  the  training  effectiveness 
of  the  devices  (and  device  features)  are  optimized 
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APPENDIX  A 

QUESTIONNAIRES  AND  STRUCTURED  INTERVIEW  PROTOCOL 
USED  DURING  THE  EVALUATION  OF  THE  AMTESS  DEVICES 


TRAINEE  QUESTIONNAIRE 


Please  answer  the  following  questions  about  the  simulator: 
1.  How  do  you  feel  about  the  simulator?  (Circle  one.) 


like  it  LIKE  IT  NO  STRONG  DISLIKE  IT  DISLIKE  IT 

VERY  MUCH  SOMEWHAT  FEELINGS  SOMEWHAT  VERY  MUCH 

ABOUT  IT 


2.  Why  did  you  answer  question  ^1  as  you  did? 


3.  What  specific  features  of  the  simulator  do  you  like  or  dislike? 


■1.  Why  do  you  like  or  dislike  these  features? 


5.  Please  make  any  additional  comments  you  feel  are  aoprocriate. 


A-1 


INITIAL  INSTRUCTOR  QUESTIONNAIRE 


Please  answer  the  following  questions  about  the  simulator: 

1.  How  easy  or  hard  is  it  to  learn  to  use  the  simulator?  (Circle  one.) 


VERY  EASY 


FAIRLY 

EASY 


AVERAGE 


FAIRLY 

hlARO 


VERY  HARD 


2.  Why  did  you  answer  question  #1  as  you  did? 


3,  How  do  you  feel  about  the  simulator?  (Circle  one.) 


LIKE  IT  LIKE  IT  NO  STRONG  DISLIKE  IT  DISLIKE  IT 

VERY  MUCH  SOMEWHAT  FEELINGS  SOMEWHAT  VERY  MUCH 

ABOUT  IT 


4.  Why  did  you  answer  question  #3  as  you  did? 


5.  Please  make  any  additional  comments  you  feel  are  appropriate. 


A-2 


INSTRUCTOR  QUESTIONNAIRE 


TASK  TO  BE  TRAINED:' _ 

#  . 

Please  answer  the  following  questions  about  the  simulator: 

1.  How  well  does  the  simulator  train  this  task?  (Circle  one.) 

VERY  MODERATELY  FMR  POORLY  VERY 

WELL  WELL  POORLY 

2.  Why  did  you  answer  question  #1  as  you  did? 

3.  What  specific  features  of  the  simulator  (involved  in  this  task) 
do  you  like  or  dislike? 

A.  Why  do  you  like  or  dislike  these  features? 


5.  Please  make  any  additional  comments  you  feel  are  appropriate. 


COURSE  DEVELOPER  QUESTIONNAIRE 


TASK  TO  BE  TRAINED: 


Please  answer  the  following  questions  about  the  simulator: 


1.  Is  this  a  critical  task  which  needs  to  be  trained?  (Circle  one.) 


YES  NO 

2.  Is  this  task  currently  trained  on  operational  equipment? 

YES  NO 

If  no,  why  not? 


3.  How  difficult  Is  it  to  perform  this  task? 

I - 1 - ^ ^ 

VERY  SOMEWHAT  ABOUT  SOMEWHAT  VERY  EASY 

DIFFICULT  DIFFICULT  AVERAGE  IN  EASY 

DIFFICULTY 


4.  At  what  skill  level  should  a  trainee  perform  this  task  at  the  end  of 
training? 


STRUCTURED  INTERVIEW  FOR  AMTESS  SIMULATORS 


NAME: _  GRADE: _ 

TITLE:  _  DATE:  _ ^ _ , 

FORT  BLISS:  _  ABERDEEN  PROVING  GROUND:  _ 

MANUFACTURER:  GRUMMAN _  SEVILLE/BURTEK _ 

SIMULATOR  CONFIGURATION: 

_  HAWK  MISSILE  TRANSMITTER 

_  NHC-250  DIESEL  ENGINE 

_ M-nOA2  STARTING/CHARGING  SYSTEM 

MOS  OF  STUDENTS  TAUGHT  ON  SIMULATOR: 

24C10 _  63W10 _  63B30  _  63030  _  63H30 

CONDUCTED.  BY:  _ 


A-5 


PURPOSE: 


This  Interview  Is  being  conducted  as  part  of  the  evaluation  of 
the  Army  Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS) 
breadboard  simulators.  The  purpose  of  the  AMTESS  study  is  to  appraise 
different  configurations  and  features  of  generic  training  devices 
addressing  various  maintenance  tasks. 

OBJECTIVE: 

The  objective  of  this  effort  Is  to  collect  qualitative  data  from 
those  who  have  expert  knowledge  of  these  devices  or  have  taught  experi¬ 
ment  students  with  the  simulators.  The  results  of  this  material  will 
then  be  reported  in  conjunction  with  other  quantitative  information. 

METHOD: 

The  Interview  will  be  conducted  on  an  Individual  basis  and  will 
require  from  one  and  one-half  to  two  hours  to  complete.  It  consists 
of  13  questions.  The  arrangement  of  the  questionnaire  Is  as  follows: 

1  through  7  address  instructional  subjects,  8  through  11  refer  to 
hardware  features  and  the  remaining  two  questions  are  general  In 
nature.  Finally,  at  the  end  of  each  question,  the  subject  matter 
expert/ instructor  will  be  asked  to  rate  the  simulator  features  for 
their  training  value  in  the  present  setting  and  their  potential 
training  value. 

VALUE: 

The  data  collected  from  these  interviews  will  be  a  portion  of  the 
qualitative  material  gathered  for  this  study.  The  Information  learned 
from  subject  matter  experts/instructors  will  be  valuable  in  the  evalua¬ 
tion  of  the  AMTESS  concept  of  generic  maintenance  trainers. 
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0  AtitoMiM  MrforMnei  ft*4b«et 
0  *«Ca«HU4  MlfuKtlQn  lnt«rt<on 
0  N^rMoey 

e  ^rvqriMnM  Ulk  letnirlot/ 
Itliont 

0  Cduinf  cioieilttiti 
0  ]-0in«niion4l  nooult 


c  3>0  aotfult 
e  CRT  fttoMck 
0  Audio  TMObtck 
0  Sound  kffocu 
0  Rrrntor 
e  Horocooy 

0  Arriflfofflont  of  ItiiOM  on 
erodfko 

0  tdltiAf  cdpdOlItty 
0  Un1«on«1  inttructor 
0  Studont  flit 
e  Student  CRT 
0  Instructor  CRT 
e  AutOMtid  prt*1ttton  chock 
0  Vioto  disc  tnd  pUytr 
0  f looey  disc  tnd  drive 
0  Cot)  instructor  sfter 
2  errors 

e  Reouest  reoest  of  lesion 
on  CRT 

0  SelNoeced 


Sevllle/iurtek 
0  3-0  mooule 
0  Student  CRT  feedbsck 
0  Instructor  CRT  feedback 
0  Htrdcooy 
0  Student  file 
0  Rroortm  irrsnqement 
0  .‘ialfunction  insertion 
0  Trouolesnoot  only 
0  Renve/reolace  caoability 
0  Editing  lysten 
0  Random  malfunction  selection 
0  Visual  projection  unit 
0  floppy  disc  and  drive 
0  Self-paced 


RCSRONSC: 


A.  Rate  the  traininq  value  of  these  features  as  a  whole. 

NEVER  USED  OR  •NONEl  MINIMAL j  SOME  I  MODERATE  1  CONSIDERABLE  I  GREAT  i  GREATEST 
unavailable  '■  I _ I  I _ _ _ I _ 


3.  Rate  the  sotential  trainiig  value  of  these  features  as  a  whole. 
none;  hinIMAL'  some;  mqoeraTE:  considerable:  GREAT!  SREATEST 


INSTRUCTIONAL 


2.  Wh«t  wtrt  tti*-1njtnjet1on*l  fttturo  of  th#  ilauiUtor  thot  not 
<op11cab1«  CO  th«  tctwol's  training  court*? 

. . . . . . . . 


0  autsMtM  p*rfor«»ne*  ftMb«ck 
0  automiua  ml  function  (nurtu 
0  Horocooy 

0  fmorMwoc  tilt  tctnoriot/ 
Itttont 

0  Eoitino  ciooeilltitt 
0  I'Oiaitntlonol  mooulo 


i»rw«w«n 

•  0  J<0  mMuU 

'  0  CRT  ftoobock 

•  0  Audio  fMdtMCk 

•  0  Sound  iffccts 

•  0  Rplnttr 

•  0  Hordcooy 

•  0  Arnnfomtnt  of  Itttont  on 

Orogrta 

’  0  Editing  etotblllty 

•  0  Univtnti  Inttruetor 

•  0  Studtnt  flit 

•  0  Studtnt  CRT 

•  0  InttPuetOP  CRT 

•  0  AutOMttd  prt-lttton  eftttk 
0  Vidto  dlte  tnd  pitytr 

•  0  Hoooy  dlte  «nd  dr'vt 

•  0  Ctll  (nttruetop  *ft*i- 

•  Z  trrort 

•  0  R«)uttt  rtpott  of  Ittton 

on  CRT 

•  0  StINptctd 


Stvl llt/lurttk 
0 J-a  aodu[t~ 

0  Studaot  CRT  fttdOtek 
0  Inttruetor  CRT  fttdbtek 
0  Htrdeooy 
0  Studtnt  flit 
0  Progria  trrtngtntnt 
0  Ho  I  function  Inttrtion 
0  Troublttnoot  only 
0  Rtmovt/rtoltct  etotolllty 
0  Editing  tyttta 
0  Rtndoa  mtifunetlon  ttltction 
0  VltutI  projection  unit 
0  rioooy  dlte  tnd  anvt 
0  Stif'ptetd 


RESPONSE 1 


3.  In  opinion,  which  ftcturos  of  tho  Itsson  prtstntatlon  hcIptP 
mikt  cht  slnwlitor  Itssons  Inttrtsting  to  tht  itudtnts? 


AutOMtt4  pwr^orMiKt 
fMObMk/htrdcapy 
.wttAM  of  lotion 
prwiontttlon 

kro-profronnod  no  1  functions 
Solf'Otctd/pocinf  of  Itfsen 
Audio/ VI  two  Is 


Sni^n 

0  5oT7-ooeod 
0  Aoouott  halp  on  CAT 
0  Ahfltot 
0  Srtonics 
0  Audio 

0  Arinttd  Mttrltl 
0  Sound  offtcts 
0  Horoeooy 
0  MOdulo 
0  Tocftnlcol  nonuols 
0  Seood  of  prttontttlon 
0  Soood  of  coinouttr 
0  CAT  foodboct 


Sovlllt/lurttk 
0  Studont  CAT  nottriil 
0  List  of  rospondor  nuroors 
0  Studont  rospondor  unit 
0  Solf-oocod 
0  visuol  projoetlon  unit 
0  Slldos 

0  Tiennietl  ntnuols 
0  3-0  nodulo 
0  Sound  iffoets 
0  Audio  slontls 
0  CAT  foodbtck 


AUAONSC: 


3.  ?Ate  M#  3otent1il  trolnlng  value  of  these  features  as  a  wnole. 
NONf:  XISIVAL  SOME!  fOOEilftTr;  CONSIDE^BLSI  GBEAT'.  GaSATEST 


INSTRUCTIONAL 


4.  Of  Ult  ItssoQi  tauftit  by  t^t  sinuUtar,  «h1ch  onts  did  you  fMl 
«^r«  otrtlcuUrly  tffoctlvt? _ 


s  Idtntify  ui«  iMton 
0 


finMfl 

0  s««  l.ntruetor  Suldtt 
for  lUtlitfs 


s«»in«/gurmi 

•  0  ^  instructor  SuldM 

*  for  Kttlnfs 


MSrONSf; 


\ 

■  j 

i 


A.  Rate  the  training  value  of  these  lessons  as  a  whole. 


.1EVES  USED  OR 
JNAVAILABLr 

:N0NEj 

1  1 

Minm 

SOHE 

MODERATE 

CONSIDERABLE  GREAT 

GREATEST 

5 

1  1  i 

2  1 

3  1 

*  1 

.5  i  fi  1 

7 

3.  Rate  the  Dotentia!  training  value  of  these  lessons  as  a  wnole. 

•ONE!  HlhihiAL  SOHE!  HCDERATI!  CQWSIDERABIEI  GREAT'  SREATtST 
2  I"!  3  ;  T  1  S  !  7 
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>■ 


:.v. 


4 


INSTRUCTIONAL 


5.  Of  thf  lessons  tiu^nt  by  the  sinuletor.  wblcb  ones  did  you  feel 
»iere  Ineffective? 


Sni^n  Stvint/gurfli 

a  IdMtlfy  the  Iciion  •  a  Ue  tnitructor  Sufdei  •  a  See  initructsr 

a  •  for  Itttinfs  •  for  Mttlnft 

•  • 


niKHll: 


A-n 


Suldet 


INSTRUCTIONAL 


6.  W«rf  tter«  •ny  ocosiloni  whtn  you  f«1t  that  tha  difficulty  laval  of 
tha  material  being  grtsented  by  the  simulator  Mas  aLove  the  students? 


0  'M<icn  occasion  one  «ny7 
0  Miot  lovol  of  stMOfit  non 
oooroerioto? 

0  hom  <Muld  you  noko  It  non 
conootiblo? 

0  facing 


rroueUtnooting 
Using  ust  aouipnont 
Lesson  nunor 
Sogntnt  nunor 
Uso  of  manuals 
Oeoration  of  oovlco 


Sovlllo/Burtot 


irouoTosnootIng 
Kenovo/nolaco 
Task  guide  nuacer 
Lesson  nunoer 
Use  of  manuals 
Oseration  of  device 


USfONSE: 
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INSTRUCTIONAL 


7.  w*r«  tfttr*  *ny  occtsilons  xhtn  you  ftlt  th«t  th#  difficulty  Itvtl 
of  Utt  nwttrlil  6t1n9  prcstnttd  by  tb«  sIruUtor  should  b«  1ncr«istd? 


0  Miicn  occiitoRt  4nd  ■A)'? 

0  iM4t  l«vtl  of  ttudoflt  nor* 
4ooroeH4(t7 

0  No*  aould  you  m*i*  it  mart 
concttiOlt? 

0  f 40109 


SESKMSE: 


Snoaitn 

0  troueltioootin9 
0  Ui1n9  ttst  oouipatnt 
0  Ltiion  nuntr 
0  St^wit  nurotr 
0  Uit  of  rninu*!! 

0  Ootrttlon  of  dtvict 


Stvlnt/Burttk 
0  rrouei*tnoot'in9 
0  R«nv*/r*pl*ci 
0  Ttik  juid*  nu«Mr 
0  Ltlion  nuiiMr 
0  Uit  of  nonutli 
0  Ootrttlon  of  Otvic* 
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HARDWARE 


Wh*t  h*rtwtr«  fttturts  of  Wt  ilnuUtar  radt  it  mopt  tfftctivt 
cnan  convtntional  training? 


0  AcetttfbilUy  of  caovononu 
0  Stftty 

0  ffnt/cott  tfftctivtntli 


SrvwMn 

0  is«  voltaa#  on  ]>0 
0  3*0  modulo 
0  frintor 
0  HiPdcooy 

0  Studoflt  car 

0  Initruetor  MT 
0  Vlooo  dlic  ond  Bliytp 
0  Hopoy  dlik  ond  dnvo 

0  ToucA  ponti 
0  kdyeeoro 
0  Comoutor 


Stvillo/SuPtok 
0  Low  volU9t  on  3-0 
0  3-0  modulo 
0  StudOflt  CKT 
0  Rotponto  Dtnol 
0  Lilt  of  rotpondor  nunpon 
0  Vliuil  projoction  unit 
0  Initruetor  CRT 
0  Initruetor  koypoird 
0  Initruetor  panol 
0  Rrlntor 
0  Horoeopy 

0  I'loppy  01  Ik  and  drivo 
0  Computor 


RCSKNSC. 


A.  Rata  tn«  training  valut  of  these  features  as  a  whole. 

NEVR  USED  OR  INONEI  MINIMAL!  SOME!  MODERATE!  CONSIDERABLE j  SREAT!  GREATEST 

•JNAVAILABLE  i  I _ I  ! _ I _  ' 

- 3 - m - 2  ;  i  I  I— ^  r-T-,  1 

3,  Rate  the  ootential  training  value  of  these  features  as  a  -nole. 

NONE  -INIMAL’  SOME-  'MODERATE!  CCNSIDERABLE!  GREATS  GREATEST 


HARDWARE 


.9.  Whit  ««re  th«  feiturt:  that  imd«  this  simulator  tasy  Tor  the 


- iiilM . 

SnaiMn 

SevlHe/Surtek 

0 

Instructor  station 

•  0 

vioM  disc  and  playtr 

Intcrvctor  CRi 

0 

Instructor  guioaeooa 

•  0 

Floopy  disa  and  drive 

•  0 

Instructor  pa.iel 

0 

Edltinq  caoaoiilty 

•  0 

CRT 

•  0 

Ranel 

0 

taliaotllty 

•  0 

CayMard 

•  0 

Prinur 

•  0 

Suida&ook 

*  0 

Disk  drive 

•  0 

Rrlntar 

•  0 

Prograaened  lessons 

•  0 

Rrintout  for  mcH  itsson 

•  0 

3*0  module 

•  0 

Student  CRT 

•  0 

Editing  systan 

•  0 

Touch  panel 

•  0 

Malfunction  list 

•  0 

]*0  module 

•  0 

Variaole  endings  on  tasks 

•  0 

Self-cneck  at  start  of  task 

•  0 

Task/activity  guides 

•  0 

Proqranwed  lessons 

•  0 

Treuoleinoetlng  exercises 

•  *  0 

Universal  instructor 

•  0 

Flexible  program 

•  0 

Editing  systee 

•  0 

Remove/rcpiaci 

• 

•  0 

Visual  projection  unit 

• 

•  0 

Student  CRT 

• 

• 

•  0 
• 

Student  responder 

lESnnSE: 


A.  Rate  the  training  value  of  these  features  as  a  ««ho1e. 


NEVER  JSED  OR 

unavailable 

iNONEI  .MINIMAL 

1  • 

SOMEj 

•^DERATE  CONSIDERABLE! 

GREAT  1 

1 

GREATEST 

0 

.  '  i  2  ! 

r7 

4 '7  "".5  1 

6  1 

7 

3.  Rate  vie  sotential  training  value  of  these  features  as  a  whole,. 
WONE  -iNi-AL  JOME'  WOOERATSl  CONSIDERABLE:  :  »EA7EST 

-1  2 - r - 1 — ^ - 5^ - r  ,  ■ 


HARDWARE 


10.  whtt  M«r*  tiif.  features  that  mde  this  slmuletar  easy  for  t.he  students 
to  operate? 


s  SCuMnt  ivMutt 
0  Conoonenu  of  t«e  3-0 
moeule 

0  SCuoent  GulotOooK 


SruwMB 
0  I'oucn  poftti 
0  Audio 
0  3-0  nodule 
0  Guideoook 


SeviHo/turtea 
0  Student  resoonder  t>ste« 
0  3-0  nodule 
0  Suldeoodk 


3.  ?ate  the  ootentlil  ;ra1ning  value  of  t.hese  features  as  a  whole. 


lONE'  Mlh I,“AL  SOME'  *«OOEi^TI'  :ONSIDE?lABl£'-  SaSAT'.  WEATEST 


HARDWARE 


11.  Whi*.  aspects  of  the  device  «*ou1d  be  sopreprlsu  for  substituting  for 
actual  eouloincnt? 


0  Ciia  of  Inttrtine  Mlfunetlom  * 
0  Morfleooy  of  ituoont  ptrforMnea  ” 
0  Sofaty  • 

0  Avoldonet  of  routin#  taiki  • 
(<-ori«ovin9  «cetii  pontli)  • 


Gru^w 
0  J'O  noauU 
0  Hardcopy 
0  Studdfit  rvcord 
0  Prodriniwd  iMtoni 
0  Gtudonc  CKT 


Sdvilld/iurf  a 

0  J-u  moauid" 

0  Hardcopy 
0  Student  record 
0  Profrtnned  laiioni 
0  Malfunction  Iniertion 
0  flexiPle  oro^rtm 
0  Student  CRT 
0  Vliual  display  unit 
0  Gripnics 


RCSKNSC: 


A.  Rate  tbe  training  value  of  these  features  as  a  whole. 

NEVE?  USED  OR  ,N0NE|  MINIMAL!  SOME!  .MODERATE!  CONSIDERABLE  I  GREAT  I  GREATEST 

unavailable  '  \  I  I  !  I  i 

3  nn  2  i  3  ;  i  ;  sn — ^ — 

3.  Sate  the  potential  training  value  of  these  features  as  a  wnole. 
none:  MINIMAL  SOME'  *«0DERATE'  CDNSIDERABLE-  GREAT'.  GREATEST 


GENERAL 


1Z.  Wh«t  types  of  probleffls  did  the  students  h«ve? 


0  0lff1culty.'fel1oti1ne  letton 
0  LKk  of  fMdUck 
e  Difficulty  eoeritlnf  device 
0  Nelfunctioni 


6r\ieen 
0  LOCK  uot 
0  3-0  nodule  breekt 
0  Touch  oenel 

e  CRT  Indicetine  errors  tMt 

node 

0  Uslnf  nenuelt 


Sevllle/Burtek 

0  Too  little  feedbeck  during 
renove/rtoleee 
0  3*0  nodule  breeks 
0  Student  responoer  penel 
0  List  of  rcsoonoer  nuMrs 
0  Using  aenuels 


kESfONSE: 
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CUtir«aa/f1«l4  unlts- 

Sfaul«(Or;SCuMnt  rttio 
In  conjunction  oitn  ictuol 
lOuipnont  or  oy  lUolf 


Snanon 

nSSTO  *IT 

0  lotuMon  U««l  /lold  units 
0  O.S.  vt.  OrfMUStlonsI 
0  OJT 


SovlUt/BurtoS 

0  530To 

0  63B30 

0  0.£.  vf.  Orgsnizitlonsi 
0  OJT 


A.  R«tt  th*  training  v«1ui  of  tJiii  slmuUtar  as  a  wfiole. 

”SavAIlIbl?  CONSIDERABLE!  GREAT]  GREATEST 


APPENDIX  B 

RAW  DATA  FROM  QUESTIONNAIRES  ADMINISTERED  DURING  EXPERIMENT 


Table  B-1 


Responses  to  the  Instructor  Questionnaire 
Administered  during  Experiment  1 


QUESTION:  How  well  does  the  simulator  train  this  task? 
RATINGS:  1  =  very  poorly 

2  =  poorly 

3  =  fair 

4  =  moderately  well 

5  =  very  well 


Task  Number  Description  Rating 


4  Remove  and  replace  oil  pump 

5  Perform  an  oil  pressure  test  using  STE/ICE 

6  Remove  and  replace  thermostat 

7  Remove  and  replace  water  pump 

8  Remove  and  replace  alternator 

9  Perform  a  DC  voltage  test  using  STE/ICE 

10  Remove  and  replace  starter  motor 

11  Perform  a  DC  current  test  using  STE/ICE 

12  Remove  and  replace  fuel  pump 

13  Perform  a  resistance  test  using  STE/ICE 

17  Troubleshoot  and  repair  engine  (oil  pump  failure 


low  pressure)  5 

18  Troubleshoot  and  repair  the  cooling  system  (thermostat 

failure-low  temperature)  4 

19  Troubleshoot  and  repair  the  cooling  system  (water  pump 

failure-high  temperature)  4 

20  Troubleshoot  and  repair  fuel  system  (fuel  pump  failure- 

engine  stalls)  4 

21  Troubleshoot  and  repair  electrical  system  (starter 

motor,  failure-slow  start)  3 

22  Troubleshoot  and  repair  electrical  system  (alternator 

failure-high  charge)  4 

23  Troubleshoot  and  repair  electrical  system  (loose 

alternator  belt-low  charge)  4 

24  Troubleshoot  and  repair  electrical  system  (alternator 

failure-BG  point  low)  4 

25  Troubleshoot  and  reoair  electrical  system  (fuel  pump 

failure-hard  start)  4 

26  Troubleshoot  and  repair  electrical  system  (battery 

switch  failure)  5 

27  Troubleshoot  and  repair  electrical  system  (front 

harness  failure)  4 

28  Troubleshoot  and  repair  NHC-250  Diesel  (protective 

control  box  failure)  4 
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Table  B-2 


Responses  to  Question  1  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  1 


QUESTION:  Is  this  «  criticti  task  which  needs  to  be  trained? 

RESPONSE:  Yes.  No 

Responses 

Task  Number  Description  Developer  1  Developer  2 


4  Remove  and  replace  oil  pump  Y 

5  Perform  an  oil  pressure  test  using 

Slt/ICE  Y 

6  Remove  and  replace  thermostat  Y 

7  Remove  and  replace  water  pump  Y 

8  Remove  and  replace  alternator  Y 

9  Perform  a  DC  voltage  test  using 

STE/ICE  Y 

10  Remove  ane  replace  starter  motor  Y 

11  Perform  a  DC  current  test  using 

STE/ICE  Y 

12  Remove  and  replace  fuel  pump  Y 

13  Perfonn  a  resistance  test  using 

STE/ICE  Y 

17  Troubleshoot  and  repair  engine 

(oil  pump  failure-low  pressure)  Y 

18  Troubleshoot  and  repair  the  cooling 

system  (thermostat  failure-low 
temperature)  Y 

19  Troubleshoot  and  repair  the  cooling 

system  (water  pump  failure-high 
temperature)  Y 

20  Troubleshoot  and  repair  electrical 

system  (fuel  pump  failure-engine 
stalls)  Y 

21  Troubleshoot  and  repair  electrical 

system  (starter  motor  failure- 

slow  start)  Y 

22  Troubleshoot  and  repair  electrical 

system  (alternator  failure-high 
charge)  Y 

23  Troubleshoot  and  repair  electrical 

system  (loose  alternator  belt-low 
charge)  Y  , 

24  Troubleshoot  and  repair  electrical 

system  (alternator  fallure-BG 

point  low)  Y 

25  Troubleshoot  and  repair  electrical 

system  (fuel  pump  failure-hard  start)  Y 

26  Troubleshoot  and  repair  electrical 

system  (battery  switch  failure)  Y 

27  Troubleshoot  and  repair  electrical 

system  (front  harness  failure)  Y 

28  Troubleshoot  and  repair  NHC-250 

Diesel  (protective  control  box 
failure) 


N 

N 

N 

N 

Y 

Y 

Y 

Y 

Y 

N 

Y 

N 

N 

Y 

Y 

Y 

N 

N 

Y 

Y 

Y 


TaMe  B-3 


Responses  to  Question  2  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  1 


QUESTION:  Is  this  task  currently  trained  on  operational  equipment? 
RESPONSE:  Us,  No 


Task  Nunber 


Responses 

Description  Developer  1  Developer  2 


4 

Rerove  and  replace  oil  pump 

Y 

y“ 

5 

Perform  an  oil  pressure  test  using 
STE/ICE 

N 

Y 

6 

Remove  and  replace  thermostat 

Y 

Y 

7 

Remove  and  replace  water  pump 

Y 

Y 

8 

Remove  and  replace  aUemator 

Y 

Y 

9 

Perform  a  DC  voltage  test  using 

STE/ICE 

Y 

Y 

10 

Remove  ane  replace  starter  motor 

Y 

Y 

11 

Perform  a  DC  current  test  using 

STE/ICE 

Y 

Y 

12 

Remove  and  replace  fuel  pump 

Y 

Y 

13 

Perform  a  resistance  test  using 

STE/ICE 

Y 

n'^ 

17 

Troubleshoot  and  repair  engine 
(oil  pump  failure-low  pressure) 

Y 

Y 

18 

Troubleshoot  and  repair  the  cooling 
system  (thermostat  failure-low 
temperature) 

Y 

Y 

19 

Troubleshoot  and  repair  the  cooling 
system  (water  pump  failure-high 
tem|)erature) 

Y 

Y 

20 

Troubleshoot  and  repair  electrical 
system  (fuel  pump  failure-engine 
stalls) 

Y 

Y 

21 

Troubleshoot  and  repair  electrical 
system  (starter  motor  failure- 
slow  start) 

Y 

Y 

22 

Troubleshoot  and  repair  electrical 
system  (alternator  failure-high 
charge) 

Y 

Y 

23 

Troubleshoot  and  repair  electrical 
system  (loose  alternator  belt-low 
charge) 

Y 

Y 

24 

Troubleshoot  and  repair  electrical 
system  (alternator  fallure-BG 
point  low) 

Y 

Y 

25 

Troubleshoot  and  repair  electrical 
system  (fuel  pump  failure-hard  start' 

Y 

Y 

26 

Troubleshoot  and  repair  electrical 
system  (battery  switch  failure) 

Y 

Y 

27 

Troubleshoot  and  repair  electrical 
system  (front  harness  failure) 

Y 

Y 

28 

Troubleshoot  and  repair  NHC-250 

Diesel  (protective  control  box 
failure) 

- 

- 

“course  developer  Indicated  that  this  task  Is  taught  on  operational  equipment  onl 
^to  support  the  AMTESS  evaluation. 

“Course  developer  Indicated  that  this  task  Is  taught  on  an  engine  other  than  the 
Cummins  NHC-250. 
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Table  B-4 


Responses  to  Question  3  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  1 


QUESTION:  How  difficult  1$  it  to  perform  this  task? 
WTINGS:  1  ■  very  easy 

2  ■  somewhat  easy 

3  •  about  average  In  difficulty 

4  ■  somewhat  difficult 

5  ■  very  difficult 


Responses 

Task  Ninfaer  Description  Developer  1  Developer  2 


4  Remove  and  replace  oil  pianp  3 

5  Perform  an  oil  pressure  test  using 

STE/ICE  3 

6  Remove  and  replace  thermostat  2 

7  Remove  and  replace  water  ptanp  4 

8  Remove  and  replace  alternator  2 

9  Perform  a  DC  voltage  test  using 

STE/ICE  3 

10  Remove  ane  replace  starter  motor  3 

11  Perform  a  DC  current  test  using 

STE/ICE  4 

12  Remove  and  replace  fuel  pump  3 

13  Perform  a  resistance  test  using 

STE/ICE  4 

17  Troubleshoot  and  repair  engine 

(oil  pump  failure-low  pressure)  4 


18  Troubleshoot  and  repair  the  cooling 

system  (thermostat  fallure-luw 
temperature)  4 

19  Troubleshoot  and  re|<a1r  the  cooling 

system  iwater  pump  failure-high 
temperature)  4 

20  Troubleshoot  and  repair  electrical 

system  (fuel  pump  failure-engine 
stalls)  4 

21  Troubleshoot  and  repair  electrical 

system  (starter  motor  failure- 

slow  start)  4 

22  Troubleshoot  and  repair  electrical 

system  (alternator  failure-high 

charge)  4 

23  Troubleshoot  and  repair  electrical 

system  (loose  alternator  belt-low 
charge)  4 

24  Troubleshoot  and  repair  electrical 

system  (alternator  fallure-BG 

point  low)  4 

25  Troubleshoot  and  repair  electrical 

system  (fuel  pump  failure-hard  start)  4 

26  Troubleshoot  and  repair  electrical 

system  (battery  switch  failure)  4 

27  Troubleshoot  and  repair  electrical 

system  (front  harness  failure)  4 

28  Troubleshoot  and  repair  NHC-250 

Diesel  (protective  control  box 
failure) 


3 

3 

2 

3 

3 

4 
4 

4 

4 

2 

2 

2 

2 

4 

4 

4 

2 

4 

4 

4 

4 
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Table  B-5 

Responses  to  Question  4  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  1 


QUESTION:  At  what  skill  level  should  a  trainee  perform  this  task  at  the 
end  of  training? 

RATINGS:  1  •  novice 

2  •  better  than  novice 

3  •  apprentice 

4  •  less  than  expert 

5  *  expert 


Responses 

Task  Nunber  Description  Developer  1  Developer  2 


4 

Rennve  and  replace  oil  pump 

3 

3 

S 

Perform  an  oil  pressure  test  using 

STE/ICE 

3 

3 

6 

Rennve  and  replace  thennostat 

4 

3 

7 

Remove  and  replace  water  punp 

3 

•  3 

8 

Remove  and  replace  alternator 

4 

4 

9 

Perform  a  DC  voltage  test  using 

STE/ICE 

3 

3 

10 

Remove  ane  replace  starter  motor 

4 

3 

11 

Perform  a  DC  current  test  using 

STE/ICE 

3 

3 

12 

Remove  and  replace  fuel  pump 

3 

13 

Perform  a  resistance  test  using 

STE/ICE 

3 

• 

17 

Troubleshoot  and  repair  engine 

(oil  pump  failure-low  pressure) 

3 

c 

18 

Troubleshoot  and  repair  the  cooling 

system  (thermostat  failure-low 
temperature) 

4 

4 

.  19 

Troubleshoot  and  repair  the  cooling 

system  (water  pump  failure-high 
temperature) 

4 

4 

20 

Troubleshoot  and  repair  electrical 

system  (fuel  pump  failure-engine 

stalls) 

4 

3 

21 

Troubleshoot  and  repair  electrical 

system  (starter  motor  failure- 
slow  start) 

4 

3 

22 

Troubleshoot  and  repair  electrical 

system  (alternator  failure-high 
charge) 

4 

3 

23 

Troubleshoot  and  repair  electrical 

system  (loose  alternator  belt-low 
charge) 

4 

4 

24 

Troubleshoot  and  repair  electrical 

system  (alternator  fallure-BG 

point  low) 

4 

J 

25 

Troubleshoot  and  repair  electrical 

system  (fuel  pimp  failure-hard  start) 

4 

3 

26 

Troubleshoot  and  repair  electrical 

system  (battery  switch  failure) 

4 

3 

27 

Troubleshoot  and  repair  electrical 

system  (front  harness  failure) 

4 

3 

28 

Troubleshoot  and  repair  NHC-250 

Diesel  (protective  control  box 
failure) 

- 

- 
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APPENDIX  C 

RAW  DATA  FROM  QUESTIONNAIRES  ADMINISTERED  DURING  EXPERIMENT  2 


Table  C-1 


Responses  to  the  Instructor  Questionnaire 
Administered  during  Experiment  2 


QUESTION:  How  well  does  the  simulator  train  this  task? 
MTINGS:  1  =  very  poorly 

2  =  poorly 

3  =  fair 

4  =  moderately  well 

5  =  very  well 


Task  Number  Description  Rating 


4  Remove  and  replace  oil  pump 

5  Perform  an  oil  pressure  test  using  STE/ICE 

6  Remove  and  replace  thermostat 

7  Remove  and  replace  water  pump 

8  Remove  and  replace  alternator 

9  Perform  a  DC  voltage  test  using  STE/ICE 

10  Remove  and  replace  starter  motor 

11  Perform  a  DC  current  test  using  STE/ICE 

12  Remove  and  replace  fuel  pump 

13  Perform  a  resistance  test  using  STE/ICE 

17  Troubleshoot  and  repair  engine  (oil  pump  failure- 


low  pressure)  5 

18  Troubleshoot  and  repair  the  cooling  system  (thermostat 

failure-low  temperature)  5 

19  Troubleshoot  and  repair  the  cooling  system  (water  pump 

failure-high  temperature)  5 

20  Troubleshoot  and  repair  fuel  system  (fuel  pump  failure- 

engine  stalls)  5 

21  Troubleshoot  and  repair  electrical  system  (starter 

motor  failure-slow  start)  5 

22  Troubleshoot  and  repair  electrical  system  (alternator 

failure-high  charge)  5 

23  Troubleshoot  and  repair  electrical  system  (loose 

alternator  belt-low  charge)  5 

24  Troubleshoot  and  repair  electrical  system  (alternator 

failure-BG  point  low)  5 

25  Troubleshoot  and  reoair  electrical  system  (fuel  pump 

failure-hard  start;  5 

26  Troubleshoot  and  repair  electrical  system  (battery 

switch  failure)  5 

27  Troubleshoot  and  repair  electrical  system  (front 

harness  failure)  5 

28  Troubleshoot  and  repair  NHC-250  Diesel  (protective 

control  box  failure)  5 
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Table  D-1 


Responses  to  the  Instructor  Questionnaire  Administered  during  Experiment  3 


QUESTION:  How  well  does  the  simulator  train  this  task? 
RATINGS:  1  =  very  poorly 

2  =  poorly 

3  =  fair 

4  =  moderately  well 

5  =  very  well 


Ratings 
Instructor  No. 


Lesson 

Segments 

1 

2 

3 

4 

5 

6 

Introduction 

0: 

Introduction,  Part  1 

4 

■5 

5 

3 

2 

3 

1: 

Introduction,  Part  2 

3 

4 

3 

2 

1 

3 

1 

2: 

VTM  Set-up  and  Check-Out  Tutorial 

5 

» 

3 

2 

2 

3: 

VTM  Set-up  and  Check-out  Exercise 

5 

5 

4 

- 

2 

2 

2 

4: 

Introduction  to  the  Starting  System 

4 

• 

4 

2 

2 

5: 

Starting  System  Problem,  Part  1 
(VTM  Set-up  and  Check-out) 

5 

5 

4 

2 

2 

2 

6: 

Starting  System  Problem,  Part  2 
(Troubleshooting  Defective  Trans¬ 
mission  Neutral  Position  Switch) 

4 

2 

1 

1 

• 

3 

7: 

Charging  System  Problem  1 
(Defective  Lead  1) 

4 

4 

3 

1 

8: 

Charging  System  Problem  2 
(Defective  Voltage  Regulator) 

3 

3 

2 

9: 

Charging  System  Problem  3 
(Defective  Generator) 

4 

4 

4 

- 

- 

- 

Table  D-2 


Responses  to  Question  1  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  3 


QUESTION  :  Is  this  a  critical  task  which  needs  to  be  trained? 
RESPONSES:  Yes,  No 


Responses 
Course  Developer 


Lesson 

Segments 

1 

2 

3 

Introduction 

0: 

Introduction,  Part  1 

. 

N 

N 

1: 

Introduction,  Part  2 

- 

Y 

Y 

1 

2: 

VTM  Set-up  and  Check-out  Tutorial 

Y 

Y 

3; 

VTM  Set-up  and  Check-out  Exercise 

- 

Y 

- 

2 

4: 

5: 

Introduction  to  the  Starting  System 
Starting  System  Problem,  Part  1 
(VTM  Set-up  and  Check-out) 

- 

Y 

Y 

Y 

Y 

- 

6: 

Starting  System  Problem,  Part  2 

(Troubleshooting  Defective  Trans¬ 
mission  Neutral  Position  Switch) 

- 

N 

N 

3 

7: 

Charging  System  Problem  1 
(Defective  Lead  1) 

Y 

Y 

8: 

Charging  System  Problem  2 
(Defective  Voltage  Regulator) 

Y 

Y 

9: 

Charging  System  Problem  3 
(Defective  Generator) 

Y 

Y 

Table  D-3 

Responses  to  Question  2  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  3 


QUESTION: 

Is  this 

task  currently  trained  on  operational 

equipment? 

RESPONSES: 

Yes,  No 

Response'^ 

Course  Developer 

Lesson 

Segments 

1 

2 

3 

Introduction  0: 

Introduction,  Part  1 

N 

N 

1: 

Introduction,  Part  2 

- 

Y 

Y 

1 

2: 

VTM  Set-up  and  Check-out  Tutorial 

•  •• 

Y 

Y 

3: 

VTM  Set-up  and  Check-out  Exercise 

- 

Y 

- 

2 

4: 

Introduction  to  the  Starting  System 

Y 

Y 

5: 

Starting  System  Problem,  Part  1 
(VTM  Set-up  and  Check-out) 

Y 

Y 

6: 

Starting  System  Problem,  Part  2 
(Troubleshooting  Defective  Trans¬ 
mission  Neutral  Position  Switch) 

• 

N 

Y 

3 

7: 

Charging  System  Problem  1 
(Defective  Lead  1) 

Y 

N 

8: 

Charging  System  Problem  2 
(Defective  Voltage  Regulator) 

Y 

Y 

9: 

Charging  System  Problem  3 
(Defective  Generator) 

■ 

N 

N 

D-3 


Table  D-4 


Responses  to  Question  3  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  3 


QUESTION:  How  difficult  is  it  to  perform  this  task? 
RATINGS:  1  =  very  easy 

2  =  somewhat  easy 

3  =  about  average  in  difficulty 

4  =  somewhat  difficult 

5  =  very  difficult 


ilesponses 
Course  Developer 


Lesson 

Segments 

1 

2 

3 

Introduction 

0: 

Introduction,  Part  1 

1 

1: 

Introduction,  Part  2 

3 

1 

3 

1 

2: 

VTM  Set-up  and  Check-out  Tutorial 

2 

3 

3: 

VTM  Set-up  and  Check-out  Exercise 

- 

1 

- 

2 

4: 

Introduction  to  the  Starting  System 

1 

2 

5: 

Starting  System  Problem,  Part  1 
(VTM  Set-up  and  Check-out) 

3 

1 

6: 

Starting  System  Problem,  Part  2 
(Troubleshooting  Defective  Trans¬ 
mission  Neutral  Position  Switch) 

2 

2 

3 

7: 

Charging  System  Problem  1 
(Defective  Lead  1) 

3 

2 

8: 

Charging  System  Problem  2 
(Defective  Voltage  Regulator) 

4 

4 

9: 

Charging  System  Problem  3 
(Defective  Generator) 

- 

2 

3 
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Table  D-5 

Responses  to  Question  4  of  the  Course  Developer  Questionnaire 
Administered  during  Experiment  3 


QUESTION:  At  what  skill  level  should  a  trainee  perform  this  task  at  the  end 
of  training? 

RATINGS:  1  =  novice 

2  =  better  than  novice 

3  =  apprentice 

4  =  less  than  expert 

5  =  expert 


Responses 
Course  Developer 


Lesson 

Segments 

1 

2 

3 

Introduction 

0: 

Introduction,  Part  1 

4 

1: 

Introduction,  Part  2 

3 

5 

2 

1 

2: 

VTM  Set-up  and  Check-out  Tutorial 

• 

5 

2 

3: 

VTM  Set-up  and  Check-out  Exercise 

- 

5 

- 

2 

4: 

Introduction  to  the  Starting  System 

• 

5 

2 

5: 

Starting  System  Problem,  Part  1 
(VTM  Set-up  and  Check-out) 

3 

5 

6: 

Starting  System  Problem,  Part  2 
(Troubleshooting  Defective  Trans¬ 
mission  Neutral  Position  Switch) 

- 

5 

1 

3 

7: 

Charging  System  Problem  1 
(Defective  Lead  1) 

3 

5 

8: 

Charging  System  Problem  2 
(Defective  Voltage  Regulator) 

• 

4 

4 

9: 

Charging  System  Problem  3 
(Defective  Generator) 

- 

3 

4 

to 
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APPENDIX  E 

RAW  DATA  FROM  QUESTIONNAIRES  ADMINISTERED  DURING  EXPERIMENT  5 
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Table  E-1 

Responses  to  the  Instructor  Questionnaire 
Administered  during  Experiment  5 


QUESTION:  How  well  does  the  simulator  train  this  task? 
RATINGS:  1  =  very  poorly 

2  =  poorly 

3  =  fair 

4  =  moderately  well 

5  =  very  well 


Content  Area  Rating 

Introduction  to  the  transmitter  4 

Master  oscillator  and  power  amplifier 
high  voltage  circuits  5 

Modulator  bias  and  arc  detection  circuits  5 

Noise  degeneration  circuits  2 

Radio  frequency  generation  circuits  5 

Modulation  circuits  4 
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